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FOREWORD 


Army  personnel  managers  and  school  house  proponents  frequently  need  to  make 
tradeoffs  between  Soldier  numbers,  quality,  and  training  to  establish  and  defend 
minimum  enlistment  standards  for  entry-level  military  occupational  specialties  (MOS). 
The  standards  must  be  set  to  reconcile  the  difficulty  of  recruiting  qualified  individuals 
with  the  effort  required  for  training  Soldiers  to  perform  successfully  in  their  assigned  job. 
When  the  standards  are  too  stringent,  then  it  will  be  difficult  to  identify  a  sufficient  pool 
of  qualified  recruits.  Conversely,  if  the  standards  are  relaxed  too  far,  then  Soldiers  may 
not  perform  adequately  in  training  or  in  their  jobs.  Often,  data  pertaining  to  the  tradeoffs 
required  to  establish  enlistment  standards  exist  but  are  not  integrated  into  the  decision 
process  because  they  are  not  readily  accessible. 

This  study  addresses  the  above  need  by  developing  and  evaluating  a  method  for 
estimating  the  effects  of  changes  in  minimum  enlistment  standards  on  academic  attrition 
rates.  The  proposed  method  can  be  applied  to  assess  the  practical  implications  of 
proposed  changes  in  enlistment  standards  on  academic  attrition  from  technical  training. 
Results  of  a  large-scale  simulation  demonstrate  the  utility  of  the  proposed  method  across 
several  different  operational  scenarios.  Recommendations  and  materials  for  applying  the 
method,  so  as  to  ensure  optimal  decisions  about  where  best  to  set  enlistment  standards  for 
technical  training,  are  provided  in  the  report. 

Both  this  study  (utilizing  simulation  methodology)  as  well  as  other  on-going  ARI 
research  (drawing  on  actual  training  performance  data)  were  undertaken  to  assist 
managers  and  proponents  with  enlistment  standards  issues.  The  results  and 
recommendations  have  been  provided  to  the  Human  Resources  Command  staff,  and 
deemed  useful  in  understanding  the  standards  setting  process. 


BARBARA  A.  BLACK 
Acting  Technical  Director 
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ESTIMATING  ACADEMIC  ATTRITION  FROM  TECHNICAL  TRAINING  SCHOOL 
DATA:  METHOD  AND  SIMULATION  RESULTS 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Army  personnel  managers  frequently  need  to  make  tradeoffs  between  Soldier  numbers, 
quality,  training  effectiveness,  and  a  host  of  other  factors  when  making  personnel  management 
and  training  decisions.  The  purpose  of  the  present  study  was  to  propose  and  demonstrate  a 
logistic  regression-based  approach  for  estimating  academic  attrition  rates.  This  approach  enables 
Army  personnel  managers  and  strategic  planners  to  evaluate  the  aforementioned  tradeoffs  when 
making  decisions  about  where  to  set  minimum  enlistment  standards. 

Method: 

This  study  applies  a  logistic  regression-based  framework  to  modeling  academic  attrition 
from  technical  training  school  data.  This  attrition  model  estimates  the  percentage  of  Soldiers 
meeting  the  minimum  enlistment  standards  that  are  not  expected  to  complete  technical  training 
for  academic  reasons.  Data  requirements  of  the  model  are  simple  and  involve  Soldier  aptitude 
scores  and  training  outcome  data  (i.e.,  pass  or  fail).  The  initial  model  estimation  and  subsequent 
attrition  rate  prediction  analyses  are  packaged  in  a  ready-to-use  statistical  program,  making  it 
possible  for  Army  managers  to  apply  the  approach  proposed  in  this  study. 

We  conducted  a  large-scale  simulation  study  to  demonstrate  and  assess  the  performance 
of  the  logistic  regression-based  approach.  Actual  training  school  data  from  a  selected  MOS  were 
employed  to  derive  an  initial  set  of  realistic  training  performance  parameters.  We  then 
alternately  modified  a  subset  of  these  parameters  to  represent  different  training  school  scenarios 
Army  managers  are  likely  to  encounter  on  the  job.  Using  known  training  requirements  and 
policy  in  MOS  schools,  we  repeatedly  generated  samples  of  synthetic  pass/fail  training  outcome 
data  from  Soldier  aptitude  scores  and  training  performance  data  simulated  under  the  different 
scenarios  and  using  different  sample  sizes.  The  logistic  regression-based  model  was  then 
separately  applied  to  the  replicated  samples  of  synthetic  training  outcome  and  aptitude  score  data 
to  estimate  attrition  rates  under  a  range  of  minimum  enlistment  standards. 

The  simulation-based  analysis  that  was  carried  out  provided  a  cost-effective  approach  for 
evaluating  the  performance  of  the  proposed  logistic  regression-based  attrition  model  under 
different  operational  school  scenarios.  Equally  important,  the  replicated  simulation  design 
enabled  us  to  assess  the  sampling  error  associated  with  the  attrition  rate  estimates  under  different 
sample  sizes  and  scenarios.  Given  real-world  sample  size  constraints,  simulation  provides  a 
more  reliable  assessment  of  the  unknown  underlying  sampling  error  compared  to  those  based  on 
statistical  formulas,  which  typically  assume  very  large  sample  sizes. 

Findings: 

The  major  findings  of  our  simulation  are  threefold.  First,  a  simple  approach  based  on  the 
logistic  regression  using  only  cognitive  aptitude  information  is  adequate  for  the  purpose  of 
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evaluating  the  impact  of  changes  in  minimum  enlistment  standards  on  academic  attrition  for 
MOS  with  medium  complexity/validity  or  greater.  Second,  the  sample  size  on  which  academic 
attrition  estimates  are  based  can  significantly  impact  the  quality  of  the  decisions  made  based  on 
these  estimates.  The  results  indicated  that  a  sufficiently  large  analysis  sample  size  allows 
smaller  changes  in  minimum  enlistment  standards  to  achieve  a  targeted  attrition  with  high 
confidence,  which  in  turn  translates  to  potential  savings  in  terms  of  the  size  of  the  eligible 
applicant  pool.  Third,  related  personnel  and  training  decisions  could  be  greatly  improved  by 
extending  the  current  model  to  incorporate  information  in  addition  to  cognitive  aptitude.  This  is 
particularly  true  for  MOS  where  aptitude  does  not  significantly  predict  performance. 

Utilization  of  Findings: 

A  number  of  practical  recommendations  for  performing  and  interpreting  this  analysis  that 
could  aid  operational  decisions  are  presented.  In  particular,  the  sample  size  analyses  and 
simulation  programs  are  directly  useful  for  planning  MOS  training  school  data  collection  in  the 
future.  The  simulation  methodology  used  in  this  study  is  also  valuable  for  assessing  the  impact 
of  alternative  training  or  testing  requirements  that  may  be  under  consideration  for  managing 
academic  attrition.  This  study  concludes  with  suggestions  for  future  research  that  could  greatly 
extend  the  proposed  approach  and  further  assist  Army  managers  when  making  the  personnel  and 
training  decisions  that  motivated  this  study. 
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INTRODUCTION 


Background 

Army  personnel  managers  frequently  need  to  make  tradeoffs  between  Soldier  numbers, 
quality,  training  effectiveness,  and  a  host  of  other  factors  when  making  personnel  management 
and  training  decisions.  However,  often  neither  the  data  nor  methods  for  effectively  evaluating 
the  data  are  easily  available  to  inform  manager’s  decisions.  This  is  problematic  given  the 
importance  of  the  personnel  decisions  managers  must  make  in  an  increasingly  dynamic  and 
complex  operating  environment,  such  as  the  Army.  There  are  several  reasons  why  these 
decisions  are  critical  to  Army  managers  and  the  Army  as  a  whole. 

First,  personnel  decisions  ultimately  determine  the  placement  and  quality  of  the  training 
tens  of  thousands  of  Soldiers  receive  each  year,  which  impacts  future  performance  and  morale. 
This  is  especially  significant  given  the  time  and  investment  the  Army  makes  in  advanced 
individual  training  (AIT)  of  recruits,  roughly  $6,100  to  $16,300  per  enlistee  (U.S.  General 
Accounting  Office,  1997),  in  addition  to  the  cost  of  basic  training.  Second,  these  decisions  are 
associated  with  major  tradeoffs  between  one  or  more  of  the  Army’s  operational  objectives.  For 
example,  while  raising  minimum  enlistment  standards  is  likely  to  produce  higher  levels  of 
(overall)  Soldier  performance,  doing  so  makes  it  substantially  more  difficult  to  meet  accession 
goals,  as  more  Soldiers  would  fail  the  standards.  Likewise,  while  lowering  enlistment  standards 
makes  it  easier  to  fulfill  accession  goals,  overall  training  quality  and  subsequent  performance 
may  suffer,  as  the  average  quality  of  students  would  be  lower.  Further,  the  tradeoffs  associated 
with  these  decisions  are  often  complicated  by  shifts  in  the  Army’s  recruiting  environment  or 
public  policy,  over  which  Army  personnel  managers  exert  little  control. 

In  summary,  understanding  the  impact  of  changes  in  minimum  enlistment  standards  on 
training  attrition  (and  other  personnel  outcomes)  is  important  for:  (a)  maintaining  (or  improving) 
overall  Soldier  performance  and  morale;  (b)  effectively  managing  Army  jobs  or  military 
occupational  specialties  (MOS)  and  the  larger  recruiting  and  training  mission;  and  (c)  identifying 
the  best  solution  for  resolving  the  critical  tradeoffs  managers  face  when  making  key  personnel 
decisions. 


Purpose  of  Report 

This  report  serves  the  following  purposes: 

1-  To  propose  and  describe  an  approach  for  estimating  academic  attrition  rates  under  different 
operational  scenarios.  Note  that  this  report  deals  exclusively  with  modeling  academic  attrition. 
Training  attrition  due  to  non-academic  reasons  (e.g.,  health)  is  not  addressed  by  the  approach 
proposed  and  described  in  this  report. 

2.  To  discuss  the  effects  of  lowering  (or  raising)  minimum  enlistment  standards  on  academic 
attrition  rates  estimated  using  the  proposed  approach.  Note  that  “minimum  enlistment 
standards”  and  “cut  scores”  are  used  interchangeably  throughout  this  report.  For  those 
unfamiliar  with  the  terminology,  “cut  scores”  are  defined  as  the  minimum  score  a  Soldier  can 
achieve  on  a  screening  test  (or  other  assessment)  to  be  eligible  for  technical  training  in  a 
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particular  MOS.  In  particular,  we  discuss  issues  (e.g,,  practical,  methodological,  and  statistical) 
related  to  interpreting  the  effects  of  raising  (lowering)  cut  scores. 

3.  To  provide  practical  recommendations  for  conducting  an  attrition  rate  analysis  using  the 
proposed  approach.  Specifically,  we  present  recommendations  regarding:  (a)  collecting  data  for 
analysis;  (b)  specifying  and  estimating  the  model;  and  (c)  inferring  the  effects  of  changes  in 
enlistment  standards  on  academic  attrition. 


METHOD 

This  study  applied  a  logistic  regression-based  approach  to  modeling  academic  attrition. 
Using  a  simulation-based  analysis  strategy  based  on  real-world  data,  we  estimated  attrition  rates 
under  different  operational  conditions  Army  personnel  managers  are  likely  to  encounter.  As  an 
overview,  this  section  is  organized  as  follows.  First,  we  describe  and  propose  a  logistic 
regression-based  model  for  estimating  academic  attrition.  Additionally,  we  consider  its 
advantages  and  disadvantages  relative  to  a  model  predicting  individual  test  scores  (or  a 
composite  of  these  scores).  Second,  we  describe  data  quality  issues  and  practical  constraints 
encountered  with  existing  Army  data.  These  issues  provide  the  basis  for  the  current  study’s 
implementation  of  a  simulation-based  approach  for  investigating  the  effects  of  changes  in  Army 
enlistment  standards  on  academic  attrition.  Third,  we  describe  the  simulation-based  approach 
employed  in  this  study.  Specifically,  we  discuss  its  advantages  and  detail  the  procedures  used 
for  estimating  key  parameters  and  attrition  rates.  Additionally,  we  describe  the  selection  of 
factors  used  to  represent  the  different  operational  scenarios  Army  managers  are  likely  to 
encounter  when  making  decisions  about  enlistment  standards  and  academic  attrition  rates.  A 
mathematical  description  of  the  structure  of  training  school  data  used  in  the  simulations  is 
provided  in  Appendix  A.  Tables  and  charts  summarizing  the  results  of  the  simulation  are 
presented  in  Appendices  B  and  C.  Accompanying  Statistical  Analysis  System  (SAS)  code  for 
replicating  our  simulation  and  applying  the  proposed  logistic  regression-based  model 
operationally  to  training  school  data  can  be  found  in  Appendices  D  and  E, 

Modeling  Approach 

The  analysis  of  the  impact  of  changes  in  minimum  enlistment  standards  (i.e.,  cut  scores) 
on  academic  attrition,  viewed  simply,  involves  the  ratio  of  the  number  of  applicants  who  are  not 
expected  to  successfully  complete  the  training  to  the  total  number  in  the  relevant  applicant  pool 
that  are  eligible  for  the  target  MOS.  Since  cut  scores  are  based  on  aptitude  area  (AA)1  scores 
that  are  positively  correlated  to  training  (and  job)  performance,  raising  the  cut  score  is  expected 
to  leave  fewer  but  higher  quality  eligible  applicants.  This  is  expected  to  yield  a  lower  training 


1  Aptitude  area  (AA)  composite  scores  are  estimates  of  future  success  in  entry-level  training  and  job  performance  in 
an  MOS.  AA  scores  are  used  operationally  by  the  Army  to  assign  recruits  to  an  MOS.  AA  composites  and  the 
Armed  Forces  Vocational  Aptitude  Battery  (ASVAB)  subtests  which  comprise  them  are  described  in  many  sources 
(Greenston,  2002;  Diaz  et  al.,  2004). 
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attrition  rate  because  applicants  who  continue  to  meet  the  new  cut  score  are  more  likely  to 
successfully  complete  the  training  compared  to  applicants  who  no  longer  meet  the  new  cut  score 
and  are  dropped  from  the  eligible  pool. 

Mathematically,  there  are  two  key  components  in  evaluating  the  impact  of  changes  in  cut 
scores  on  academic  attrition  rates.  The  first  component  is  a  model  describing  the  relationship 
between  the  AA  score  of  an  applicant  and  the  probability  that  he  or  she  will  not  successfully 
complete  the  training.  The  second  component  is  the  density  or  distribution  of  the  number  of 
applicants  at  various  levels  of  AA  scores  in  the  relevant  applicant  pool.  These  ideas  are 
represented  in  the  equation  below,  which  describes  the  attrition  rate  A(C)  for  the  eligible 
applicant  pool  as  function  of  the  cut  score  C. 


A(C)  - 

\f{x)dx 

C 

The  first  component,  represented  by  the  function  P(X),  and  the  second  component,  represented 
by  the  density  f(X),  together  determine  the  numerator,  which  corresponds  to  the  count  of 
applicants  that  are  expected  to  not  successfully  complete  the  training  for  a  given  cut  score.  The 
denominator,  which  corresponds  to  the  total  number  of  eligible  applicants,  is  determined  entirely 
by  the  density  f(X)  for  a  given  cut  score.  This  equation  can  be  interpreted  as  the  average  of  the 
probability  of  attrition  weighted  by  the  density  of  the  AA  scores  that  are  above  the  minimum  cut 
score  C.  The  density  f(X)  representing  the  applicant  pool  and  P(X)  are  described  in  the  next  two 
sections. 

Applicant  Pool  Density 

The  two  natural  choices  for  the  applicant  pool  that  are  relevant  to  the  analysis  of  the 
relationship  between  cut  score  and  training  attrition  are  the  youth  population  and  the  Army 
accession  (or  input)  population.  Technically,  the  choice  of  reference  population  matters  as  the 
shape  of  the  density  f(X)  determines  the  underlying  weights  employed  in  the  computation  of  the 
attrition  rate.  For  practical  purposes,  however,  it  does  not  appear  that  the  choice  between  these 
two  reference  populations  substantially  impacts  the  overall  attrition  rates  on  the  range  of  cut 
scores  that  are  likely  of  interest  to  Army  managers.  The  rationale  for  this  is  as  follows. 

The  Army  accession  distribution  is  truncated  on  the  lower  end  and,  to  lesser  extent,  on 
the  upper  end  relative  to  the  youth  population  distribution.  The  form  of  the  attrition  formula 
A(C)  above,  however,  ignores  the  part  of  the  distribution  to  the  left  of  the  cut  score  C.  Cut 
scores  that  are  operationally  important  are  expected  to  be  within  one  standard  deviation  of  the 
youth  population  average  (see  following  section  for  a  discussion  of  cut  score  range).  Practically, 
this  means  that  the  left  end  of  the  distribution  where  the  youth  population  and  Army  accession 
distributions  differ  is  ignored  in  the  computation  of  the  attrition  rates.  This  is  not  the  case  with 
the  upper  end,  which  is  included  in  the  computation  of  attrition  rates.  However,  differences  in 
the  upper  range  of  the  youth  and  Army  accession  populations  are  not  expected  to  adversely 
impact  computed  attrition  fates  because  these  differences  are  not  as  large  as  those  in  the  lower 
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range.  Additionally,  this  is  the  part  of  the  distribution  where  the  density  f(X)  is  small,  and  hence, 
does  not  carry  significant  weight  in  the  computation  of  the  attrition.  In  summary,  differences  in  ’ 
the  youth  and  Army  accession  populations  will  not  affect  attrition  rate  computations  to  a 
meaningful  extent. 

Because  of  the  small  impact  of  differences  between  the  youth  and  Army  accession 
populations  on  the  expected  attrition  rate,  the  choice  between  them  is  best  guided  by  practical 
considerations.  For  example,  changes  in  the  Army’s  recruiting  environment  could  make  the 
truncation  on  the  high  end  of  the  youth  population  more  substantial  than  typically  observed. 

This  would  be  the  case  when  the  economy  is  strong  and  the  job  market  easily  absorbs  greater 
numbers  of  high  quality  youth.  In  this  case,  an  Army  accession  reference  population  might  be 
tailored  using  data  from  previous  recruiting  years  that  would  be  representative  of  a  strong  job 
market.  Using  this  particular  reference  population,  one  would  obtain  slightly  higher  attrition 
rates,  as  high  quality  recruits  would  be  less  represented  (i.e.,  carry  less  weight)  in  the 
computation  of  training  attrition.  Therefore,  there  may  be  cases  where  one  population  is 
preferable  because  modeling  that  specific  population  produces  estimates  of  attrition  that  more 
strongly  reflect  the  current  operational  environment  in  which  decisions  about  where  to  set 
enlistment  standards  will  be  made. 

In  our  approach,  we  use  a  youth  population-based  normal  distribution  for  f(X)  in  the 
simulation  for  convenience  and  because  there  was  not  a  particular  operational  environment  of 
special  interest.  The  approach  and  computer  program  for  estimating  attrition  rates  operationally 
(see  Appendix  D)  can  easily  be  adapted  using  any  reference  population.2 


Probability  of  Training  Attrition 

The  second  component  in  the  computation  of  A(C)  is  a  function  P(X)  describing  the 
relationship  between  training  attrition  rate  and  AA  score  of  an  applicant.  The  function  P(X)  is 
the  probability  that  an  applicant  with  AA  score  equal  to  X  will  not  successfully  complete 
training.  That  is,  P(X)  evaluates  to  a  value  between  zero  and  one  that  represents  the  percentage 
of  individuals  with  AA  score  X  that  are  expected  to  academically  attrit. 

Note  that  in  our  overall  framework,  the  complete  “cut  score  attrition  model”  is  A(C)  and 
not  P(X),  The  latter  is  the  conditional  probability  of  attrition  given  the  applicant’s  AA  score.  On 
the  other  hand,  the  cut  score  attrition  rate  A(C)  averages  the  conditional  probabilities  P(X) 
evaluated  in  the  AA  score  range  above  the  cut  score  C,  using  the  appropriate  applicant  pool 
density  f(X)  as  weights.  The  function  A(C)  evaluates  to  the  probability  that  applicants  with  AA 
score  equal  to  or  better  than  C  will  not  successfully  complete  training. 


The  averaging  used  in  the  computation  of  attrition  in  the  SAS  program  is  based  on  applicant  pool  quantiles  rather 
than  a  numerical  integration  based  on  the  specific  form  of  f(X),  for  example.  The  quantile-based  approach  is  flexible 
.as  these  may  be  computed  from  a  large  enough  data  base  representing  the  “target”  applicant  pool;  the  exact  form  of 
f(X)  need  not  be  known. 
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Training  Outcome  Variable 

The  exact  form  of  P(X)  is  dependent  on  the  type  of  “outcome”  variable  used  in  modeling 
attrition.  The  traditional  outcome  variable  used  in  this  type  of  analysis  is  the  simple  “dummy”  or 
binary  variable  taking  the  value  “1”  to  indicate  that  the  applicant  did  not  complete  training  or  “0” 
to  indicate  that  the  applicant  completed  training.  In  the  MOS  school  data  that  we  used  in  this 
study,  this  dummy  variable  was  constructed  from  a  “student  output”  code  that  indicates  whether 
or  not  the  student  graduated  from  training.  There  is  also  a  “reason”  code  in  the  data  that  can  be 
used  to  classify  non-completion  into  academic  or  non-academic  attrition.  Students  that  were 
-  -  classified  as  non-academic  attritions  were  excluded  from  our  analysis  data. 

.  .  In  this  study,  we  also  learned  about  the  structure  of  MOS  school  training  requirements 

and  testing  process  that  are  relevant  in  determining  applicant  passing  or  failing.  A  more  detailed 
technical  description  of  the  process  is  provided  in  Appendix  A.  Generally,  students  take  a  series 
of  tests  that  can  be  characterized  as  a  multiple  hurdle  system.  The  scores  on  the  tests  that  a 
,  student  took  represent  a  multivariate  outcome  variable  that  provides  more  detailed  information 
relevant  to  attrition  analysis.  This  multivariate  outcome  vector  of  test  scores  and  the  knowledge 
of  the  underlying  attrition  “generation  mechanism”  is  useful  in  defining  a  form  of  P(X)  that  is 
directly  related  to  the  underlying  process.  However,  it  is  a  more  involved  approach  using  a 
multivariate  type  of  analysis  that  we  believe  is  not  suitable  for  routine  calculations.  The  amount 
and  costs  in  terms  of  data  quality  requirements  in  a  multivariate  test  scores-based  approach  are 
also  far  greater  than  the  binary  attrition  variable  approach. 

The  approach  to  modeling  P(X)  employed  in  this  study  is  based  on  the  binary  or  dummy 
pass/fail  outcome  variable.  However,  for  the  simulation  experiments  that  were  carried  out,  we 
generated  the  pass/fail  outcome  observations  using  a  test  score-based  pass/fail  mechanism 
implementing  known  school  testing  policy  relevant  to  training  attrition. 


Logistic  Regression  Model  of  Attrition  •  -  • 

We  used  the  logistic  regression  of  the  pass/fail  outcome  observation  on  applicant  AA 
score  to  specify  P(X).  Formally,  the  relationship  between  applicant  attrition  probability  and  AA 
score  that  is  equal  to  X  is 


l  +  exp(a+6X) 


-which. is  parameterized  by.  some  unknown  constants  a  and b.  The  function  P(X)  is  defined 
between  zero  and  one  and  is  inversely  related  to  the  AA  score.  The  value  of  P(X)  decreases  to 
zero  as  AA  score  increases  and  it  increases  to  one  as  AA  score  decreases. 


There  are  several  advantages  to  a  logistic  regression-based  approach.  First,  logistic 
regression  permits  the  modeling  of  nonlinear  relationships  between  relevant  predictor  variables 
and  the  dependent  variable  of  interest  (Cohen,  Cohen,  West,  &  Aiken,  2003;  Tabachnick  & 
Fidell,  1996),  From  an  operational  perspective,  it  is  not  unreasonable  to  expect  that  many  of  the 
, .  predictors  of  academic  attrition  (e.g.,  cognitive  aptitude,  education,. disciplinary  problems,  .etc.) 
- are  nonlinearly  related  to  attrition.  For  example,  for  those  high  in  cognitive  aptitude  the 
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relationship  between  aptitude  and  attrition  is  likely  to  be  flat  as  few  high  aptitude  individuals  fail 
to  complete  training  and  the  expected  probability  for  success  is  uniformly  high.  Conversely,  for 
those  low  or  average  in  cognitive  aptitude  the  relationship  is  likely  to  be  positive,  particularly  for 
those  in  the  middle  range  of  aptitude,  as  there  is  greater  variability  in  the  proportion  of  Soldiers 
failing  to  complete  training  and  the  probability  of  success  varies  measurably  with  level  of 
cognitive  aptitude  (e.g.,  the  higher  one’s  standing  on  cognitive  aptitude,  the  higher  his/her 
probability  of  successfully  completing  training). 

An  additional  advantage  to  modeling  potential  nonlinear  relationships  is  that  a  logistic 
regression  approach  will  do  a  better  job  of  predicting  academic  attrition  across  the  full  range  of 
cognitive  aptitude,  including  those  in  the  extremes  (e.g.,  low  or  high  in  cognitive  aptitude), 
compared  to  a  linear  probability  model  (LPM)  obtained  using  ordinary  least  squares  (OLS) 
regression.  This  is  especially  important  since  operationally,  it  is  those  individuals  in  the 
extremes,  either  the  low  or  high  end  of  cognitive  aptitude,  who  will  exhibit  the  greatest 
probability  of  attriting  or  not  attriting,  respectively.  Under  LPM,  predictive  accuracy  is  weakest 
for  cases  in  the  extremes  (Cohen  et  al.,  2003;  Guion,  1998;  Tabachnick  &  Fidell,  1996), 
precisely  the  cases  the  Army  is  most  interested  in.  Therefore,  the  ability  to  model  nonlinear 
relationships  is  important  both  for  understanding  the  nature  of  the  relationship  between  Soldier 
characteristics  (e.g.,  cognitive  aptitude,  education,  etc.)  and  academic  attrition,  and  more 
operationally,  for  accurately  predicting  and  estimating  attrition  rates. 

When  compared  to  a  multivariate  test  scores-based  approach  to  modeling  P(X),  an 
advantage  to  the  logistic  regression  approach  is  that  it  simplifies  interpretation  by  aggregating 
multiple  test  scores  into  a  single,  meaningful  outcome  variable  of  interest,  academic  attrition. 
From  an  operational  perspective,  the  Army  is  ultimately  interested  in  predicting  and  explaining 
attrition,  not  necessarily  trainee  performance  on  specific  test(s),  which  vary  from  school  to 
school.  Having  a  single,  overall  outcome  variable,  as  opposed  to  ten  or  more  different  variables, 
facilitates  the  analysis.  Importantly,  it  increases  the  ease  with  which  Army  personnel  managers 
can  successfully  implement  any  proposed  methodology  for  assessing  changes  m  enlistment 
standards  on  attrition  rates  in  the  field. 

In  summary,  the  advantages  of  a  logistic  regression  approach  recommend  it  over  a  model 
focused  on  individual  test  scores.  Because  of  the  model’s  advantages,  and  its  fit  with  the  Army’s 
operational  goals,  it  is  the  preferred  approach  for  modeling  academic  attrition.  An  alternative  to 
the  logistic  model  for  specifying  P(X)  is  the  probit  model.  These  two  models,  however,  are 
practically  equivalent. 


Issues  Related  to  Modeling  Attrition 

We  have  identified  three  potential  issues  related  to  our  modeling  of  P(X).  We  describe 
each  of  these  in  turn. 

Criterion  Contamination.  The  strongest  advantage  of  the  logistic  model  approach  (or 
equivalent  binary  model)  is  its  simplification  of  the  underlying  attrition-generation  process,  in 
this  case  involving  at  least  10  to  as  many  as  30  test  scores,  into  a  single,  binary  outcome  variable 
indicating  pass/fail.  This  simplification  of  the  underlying  relevant  outcome,  however,  has  its 
pitfalls.  Specifically,  in  working  with  the  “student  output”  variable  in  our  data,  we  came  across 
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cases  where  a  student  was  classified  as  “graduate”  (i.e.,  not  attriting)  but  the  test  scores  indicated 
that  he  or  she  failed  at  least  one  test  twice;  Technically,  failing  a  test  twice  means  that  a  person 
cannot  complete  training.  We  were  informed  that  this  type  of  “exemption,”  which  is  a  form  of 
criterion  contamination,  is  in  fact  possible.  For  the  purpose  of  this  analysis,  which  is  concerned 
specifically  with  academic  attrition,  an  applicant  was  classified  as  an  attrition  if  he  or  she  did  not 
pass  a  test  after  two  attempts.  Because  “exemptions”  failed  a  test  twice,  these  cases  were 
classified  as  attritions.  This  was  done  for  two  reasons. 

First,  we  did  this  to  be  consistent  with  official  school  policy.  Officially,  school  sponsors 
and  Army  personnel  managers  define  academic  attritions  as  those  recruits  who  do  not 
successfully  pass  all  tests  required  for  technical  training.  According  to  school  policy,  recruits  are 
permitted  a  maximum  of  two  attempts  to  pass  a  required  test.  Should  a  recruit  fail  to  pass  a  test 
on  the  second  attempt,  he  or  she  technically  cannot  complete  training  without  some  form  of 
remedial  action.  Therefore,  the  practice  of  classifying  “exemptions”  as  attritions  is  consistent 
with  official  school  policy  and,  more  practically,  will  yield  the  most  accurate  assessments  of  the 
effects  of  changing  enlistment  standards  on  academic  attritions.  That  is,  while  estimates  may  be 
higher  than  actual  attrition  rates,  owing  to  the  fact  that  “exemptions”  were  excluded  during  the 
estimation  process,  the  estimates  produced  will  more  accurately  reflect  the  actual  relationship 
between  enlistment  standards  and  attrition.  Classifying  “exemptions”  as  not  attriting,  or 
excluding  them  from  the  analysis  entirely,  would  produce  inaccurate  estimates  of  the  effects  of 
changing  enlistment  standards  because  doing  so  changes  the  underlying  enlistment  standards- 
attrition  relationship.  Put  simply,  classifying  “exemptions”  as  attritions  removes  “noise”  that 
would  otherwise  cloud  the  actual  connection  between  enlistment  standards  and  academic 
attrition.  It  is  that  actual  connection  which  guides  and  informs  official  school  policy  and 
personnel  decision-making. 

The  second  reason  for  classifying  “exemptions”  as  attritions,  not  unrelated  to  the  first,  is 
that  there  is  likely  to  be  insufficient  data  to  model  “exemptions.”  Specifically,  the  reasons  for  an 
exemption  are  not  recorded  or  easily  accessible  from  technical  training  data.  In  addition, 
“exemptions”  are  not  expected  to  be  meaningfully  related  to  enlistment  standards  or  other 
variables  (e.g.,  demographics)  readily  available  to  school  sponsors  and  Army  personnel 
managers  running  an  attrition  rates  analysis.  More  importantly,  the  reasons  for  an  “exemption” 
are  not  likely  to  be  systematic,  but  random;  varying  between-recruits,  between-classes,  and 
probably  both  (e.g.,  between-recruits  within  the  same  class),  in  non-systematic  ways.  Taken 
together,  this  means  that  “exemptions”  cannot  be  alternatively  classified  (or  modeled)  in  reliable 
ways  other  than  as  “attritions”  without  introducing  some  amount  of  bias  into  the  estimation 
process;.; If  interested,  Army school  proponents  and  personnel  managers  may  adjust  estimated 
attrition  rates  after  the  fact  to  take  into  account  expected  exemptions.  However,  we  would 
generally  advise  against  this  practice,  as  meaningful  data  for  making  such  adjustments  is  not 
likely  to  be  easily  accessible. 

Incidental  Truncation.  A  second  potential  issue  related  to  modeling  P(X)  is  incidental 
truncation.  This  typically  occurs  in  non-experimental  studies  (i.e.,  using  non-randomized 
samples)  where  the  underlying  selection  mechanism  is  not  random  with  respect  to  the  outcome 
variable.  It  can  be  an  issue  in  modeling  P(X)  as  factors  influencing  a  student’s  assignment 
(selection)  into’ an  MOS,  such  as  education,  other  demographics,  or  incentives,1  can  be  associated 
with  academic  attrition.  For  example,  a  student  with  at  least  high  school  diploma  might  have  an 
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easier  time  adapting  to  the  formal  training  environment  in  an  MOS  school.  In  the  case  of 
enlistment  bonus  incentive,  which  is  typically  tied  to  a  recruit’s  completion  of  the  term  of 
service,  a  student  who  contracted  to  receive  a  high  bonus  might  have  an  extra  motivation  to 
successfully  complete  the  training.  This  hypothesis  is  significant  as  the  bonus  amount  varies 
across  MOS  and  is  designed  to  partly  influence  an  applicant’s  selection  of  MOS  in  a  way 
consistent  with  Army  priority. 

Note  that  the  use  of  an  AA  cut  score  to  determine  eligibility  is  not  a  reason  for  incidental 
truncation  in  our  problem  as  the  model  P(X)  is  already  conditional  on  the  AA  score  (X).  Note 
also  that  in  this  issue  we  are  concerned  with  the  association  between  factors  influencing 
assignment  into  an  MOS  versus  the  likelihood  of  attrition  that  is  unexplained  by  the  AA  score 
(i.e.,  the  “residual”  attrition  conditional  on  X).  In  particular,  the  more  a  factor  is  highly 
correlated  with  the  AA  score  the  less  likely  it  is  to  contribute  to  incidental  truncation. 

Formally,  incidental  truncation  is  generally  handled  by  incorporating  the  selection 
process  in  the  modeling  of  the  outcome  variable.  For  example,  in  the  linear  regression  model 
case  with  continuous  outcome  variable,  a  two-step  estimation  procedure  (Heckman,  1979)  is 
commonly  carried  out  to  handle  incidental  truncation.  In  the  first  step,  a  probit  model 
representing  the  selection  process  is  estimated,  while  the  second  step  carries  out  the  least-squares 
estimation  for  the  substantive  model  that  incorporates  a  selection  component  (see  also  Greene, 
1997,  pp.  975-978).  This  approach  or  an  alternative  maximum  likelihood  method  involves 
jointly  modeling  a  binary  variable  indicating  selection  and  a  continuous  outcome  variable  that  is 
essentially  censored  for  individuals  that  are  not  selected.  Note  that  the  “full  sample”  used  for 
carrying  out  the  estimation  in  this  method  includes  individuals  excluded  from  the  selection 
sample  (i.e.,  entire  sample  before  selection). 

In  our  problem,  the  potential  incidental  truncation  issue  is  similar  except  that  we  are  also 
dealing  with  a  binary  outcome  variable  that  is  observed  only  for  individuals  that  are  selected  into 
the  MOS.  In  this  context,  the  estimation  of  P(X)  can  be  carried  out  using  a  “censored  bivariate 
probit”  model  (Boyes,  Hoffman,  &  Low,  1989)  in  which  the  binary  outcome  variable 
representing  attrition  Is  censored  if  the  selection  binary  variable  equals  0  (i.e.,  not  selected  into 
the  MOS).  Using  this  estimation  framework  requires  switching  from  the  logit  to  the  probit 
model  for  P(X).3  More  importantly,  however,  the  full  sample  required  for  carrying  out  estimation 
of  P(X)  that  incorporates  incidental  truncation  includes  students  not  selected  into  the  MOS  under 
consideration.  This  full  analysis  sample  can  be  Army  accessions  corresponding  to  the  fiscal  years 
represented  by  students  in  the  MOS  school.  These  data  were  not  available  in  this  study,  Boyes  et 
al.  (1989)  provide  the  maximum  likelihood  equations  necessary  for  carrying  out  the  estimation. 

Mixture  of  Old  and  New  AA  Scores.  A  third  potential  issue  in  the  implementation  of  the 
proposed  method  is  the  mixture  of  old  AA  scores  based  on  nine  ASVAB  subtests  (prior  to  FY 
2002)  and  new  AA  scores  based  on  seven  ASVAB  subtests.  Although  this  may  be  less  of  an 
issue  for  MOS  with  large  numbers  of  trainees,  particularly  as  the  Army  moves  past  FY  2002, 
there  are  still  likely  to  be  cases  where  combining  school  data  using  old  and  new  AA  scores  to 


3  An  analog  to  the  bivariate  probit  is  the  bivariate  logistic  model  (McCullagh  &  Nelder,  1983),  where  both  the 
outcome  and  selection  indicator  variables  are  marginally  represented  by  separate  logistic  model. 
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achieve  a  desired  sample  size  is  necessary.  Likewise,  this  conversion  is  definitely  recommended 
if  one  is  estimating  rates  from  historical  data  for  purposes  of  establishing  baseline(s)  by  which  to 
compare  expected  rates.  For  handling  this  issue  we  propose  converting  the  old  AA  scores  into 
the  equivalent  new  AA  scores.  The  new  AA  score  should  then  be  employed  in  the  remainder  of 
the  analysis  (e.g.,  for  both  P(X)  and  A(C)  equations).  Note  that  conversion  of  the  old  AA  scores 
is  only  an  issue  if  the  original  scores  on  nine  ASVAB  subtests  are  no  longer  available.  The 
procedure  for  converting  old  AA  to  new  AA  scores  is  documented  in  Appendix  A.  For  purposes 
of  evaluating  the  proposed  method,  no  conversions  were  necessary. 

Data  Collection 

The  Army  Training  Support  Center  (ATSC)  at  Ft.  Eustis,  VA  provided  ARI  with  training 
school  data  for  building  and  testing  a  model  of  academic  attrition.  Data  for  four  MOS  roughly 
covering  the  last  two  fiscal  years  were  extracted  from  ATSC’S  Automated  Instructional 
Management  System  (AIMS)  database.  The  MOS  selected  and  for  which  training  performance 
data  were  available  represent  a  variety  of  jobs  found  in  the  Army.  The  dataset  for  each  MOS 
consisted  of  data  from  multiple  classes. 

For  the  purposes  of  the  present  study,  a  simulation-based  analysis  strategy  was  adopted. 
In  brief,  this  strategy  uses  actual  MOS  data  to  produce  computer-generated  or  synthetic  samples 
that  serve  as  input  to  the  estimation  of  attrition  rates  and  the  analysis  of  changes  in  these  rates 
from  adjustments  in  enlistment  standards.  It  should  be  emphasized  that  simulations  are  not 
intended  to  be  a  substitute  for  real-world  data.  Operational  decisions  are  best  based  on  the 
analysis  of  real-world  data.  However,  a  simulation  was  preferable  for  purposes  of  testing  and 
evaluating  the  proposed  method,  which  was  the  primary  goal  of  the  current  study,  for  the 
following  reasons. 

First,  as  the  primary  goal  of  the  current  study  is  to  propose  and  evaluate  a  method  for 
estimating  academic  attrition  rates,  a  simulation-based  analysis  strategy  is  preferred.  Since  the 
proposed  method  is  intended  for  operational  use,  it  is  critical  that  the  performance  of  the  method 
be  evaluated  as  comprehensively  and  accurately  as  possible.  A  simulation-based  approach  best 
meets  these  requirements.  By  taking  a  simulation-based  approach,  we  could  ensure  that  the 
results  from  evaluating  the  proposed  method  were  not  dependent  on  idiosyncrasies  present  in  the 
available  data.  This  enabled  us  to  evaluate  the  performance  of  the  proposed  method  as 
accurately  as  possible.  For  example,  across  three  of  the  four  MOS,  the  available  sample  size  was 
generally  small.  Small  sample  size  is  problematic,  as  it  is  associated  with  higher  levels  of 
sampling  error,- which  adversely  affects  the  accuracy  and  precision  of  estimates  (Cohen  et  al., 
2003;  Hunter  &  Schmidt,  1990);  Because  of  this,  using  the  available  data  was  likely  to  produce 
inaccurate  and  potentially  biased  estimates,  thereby  making  it  difficult  to  accurately  evaluate  the 
proposed  method. 

Second,  and  more  importantly,  by  using  the  available  data,  we  would  be  unable  to  assess 
the  impact  of  sampling  error  on  estimates  produced  by  the  proposed  method,  information  that 
would  be  extremely  useful  when  applying  the  method  in  the  field.  A  simulation-based  approach 
would  be  advantageous  in  this  regard,  because  it  would  permit  us  to  model  the  adverse  effects  of 
sampling  eiror  ;on; the  accuracy ;  of  attrition  rates  computed  operationally.  With  this  information 
we  could  provide  guidance  on  how  best  to  interpret  these  estimates  when  making  actual 
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decisions.  For  example,  with  this  information,  we  could  offer  recommendations  regarding  when 
to  make  adjustments  to  enlistment  standards  and  the  magnitude  of  the  adjustment  needed  to 
achieve  a  desired  level  of  attrition.  While  there  are  computational  formulas  for  estimating 
sampling  error,  these  formulas  typically  assume  very  large  sample  sizes,  sample  sizes  greater 
than  those  of  the  available  MOS,  Therefore,  a  simulation-based  approach  would  provide  the 
most  effective  means  for  assessing  the  impact  of  sampling  error  on  attrition  rates  estimated  under 
the  proposed  method,  and  its  implications  for  decision-making. 

From  an  operational  perspective,  a  simulation-based  approach  would  be  advantageous  for 
twe  additional  reasons.  First,  a  simulation-based  approach  would  permit  the  estimation  of 
attrition  rates  under  different  operational  scenarios;  estimates  that  would  be  impossible  to  model 
given  that  the  existing  data  were  not  collected  with  these  particular  scenarios  in  mind.  Second, 
when  done  right  and  in  operational  environments  like  the  Army’s,  simulations  provide  a  cost- 
effective  means  for  studying  different  combinations  of  variables  that  might  otherwise  take 
multiple,  individual  studies  (and  a  large  amount  of  Army  resources)  to  compile.  Conducting 
simulations  saves  valuable  time  and  critical  operational  resources.  More  importantly,  the 
information  collected  from  simulations  can  greatly  optimize  the  design  of  follow-up  studies 
using  real-world  data.  By  identifying  the  variables  most  relevant  to  the  outcome(s)  of  interest, 
researchers  can  best  maximize  the  gains,  while  minimizing  the  costs,  of  future  research  efforts. 


Simulation-Based  Analysis  Strategy 
Overview  and  General  Description 

To  assess  the  proposed  approach  for  evaluating  changes  in  minimum  enlistment 
standards  on  academic  attrition,  we  employed  a  simulation-based  analysis  strategy  that  involved 
the  following  three  major  steps. 

1 .  To  ensure  that  our  findings  would  approximate  results  from  real-world  data,  we 
selected  an  actual  MOS  whose  empirically  estimated  parameters  would  serve  as  input 
for  constructing  the  computer-generated  samples  for  use  in  the  simulation.  In 
selecting  the  MOS,  care  was  taken  to  identify  the  MOS:  (a)  whose  data  would 
provide  the  most  accurate  estimates;  and  (b)  whose  estimated  attrition  rate  was 
comparable  to  the  attrition  rate  typically  observed  in  the  Army  (e.g.,  between  10% 
and  20%).  After  identifying  the  MOS  that  best  met  these  criteria,  we  computed  the 
relevant  parameters  from  which  the  computer-generated  samples  would  be  based. 

2.  Based  on  these  real-world  parameters,  we  generated  synthetic  samples  of  varying 
sample  sizes  (, N)  that  approximated  relevant  properties  of  the  different  MOS  found  in 
the  Army.  Each  sample  consisted  of  synthetic  AA  composite  and  training 
performance  data  for  N  trainees.  To  simulate  samples  that  reflected  the  different 
operational  scenarios  Army  personnel  managers  were  likely  to  encounter  in  the  field, 
we  varied  the  parameters  from  which  the  synthetic  samples  were  generated. 

Technical  notes  on  how  AA  composite  and  training  performance  data  were  simulated 
can  be  found  in  Appendix  A.  From  the  synthetic  training  performance  data,  we 
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computed  a  dichotomous  attrition  variable  indicating  if  a  trainee  did  or  did  not  pass 
training.4 

3.  Using  the  synthetic  samples  generated  in  Step  2,  we  estimated  academic  attrition  rates 
(and  their  corresponding  standard  errors)  associated  with  varying  cut  scores.  These 
estimates  were  based  on  regression  parameters  obtained  by  applying  the  logistic 
regression-based  model  to  the  synthetic  AA  composite-attrition  data  (produced  in 
Step  2),  and  aggregating  estimates  across  samples  representing  the  same  N  and 
operational  scenario.  From  this,  we  were  able  to  infer  the  effects  of  changing 
'enlistment  standards  on  academic  attrition  under  the  different  operational  scenarios 
Army  personnel  managers  were  likely  to  encounter  on  the  job. 

Simulations  based  on  real-world  parameters,  such  as  the  approach  employed  in  the  current  study, 
have  been  widely  used  in  Army  research  and  other  social  sciences,  such  as  applied  psychology, 
economics,  and  public  policy.  For  example,  comparable  simulation-based  approaches  have  been 
successfully  used  to  identify  strategies  for  maximizing  the  efficiency  of  Army  classification 
(e.g.,  Johnson,  Zeidner,  &  Leaman,  1992;  Scholarios,  Johnson,  &  Zeidner,  1994;  Statman,  1993; 
Zeidner,  Johnson,  Vladimirsky,  &  Weldon,  2000)  and  within  applied  psychology,  to  identify 
combinations  of  different  predictors  that  maximize  both  job  (or  training)  performance  and 
demographic  representation  (e.g.,  Bobko,  Roth,  &  Potosky,  1999;  Sackett  &  Roth,  1996; 

Schmitt,  Rogers,  Chan,  Sheppard,  &  Jennings,  1997).  While  simulations,  including  that  of  the 
current  study,  are  not  intended  to  be  substitutes  for  real-world  data,  as  mentioned  in  the  previous 
section  they  do  offer  advantages,  particularly  when  evaluating  an  estimation  method. 

This  section  is  organized  as  follows.  First,  we  detail  the  selection  of  the  MOS  used  for 
producing  the  computer-generated  samples  for  the  simulation,  and  the  procedure  for  obtaining 
the  real-world  parameters  on  which  these  samples  were  based.  Second,  we  discuss  the 
construction  of  the  different  operational  scenarios  under  which  attrition  rates  would  be  estimated 
and  the  procedure  for  generating  the  synthetic  samples  for  use  in  estimating  academic  attrition. 
Third,,  we  descrihe. the  procedure  for  estimating  academic  attrition  rates  (and  accompanying 
standard  errors)  under  varying  cut  scores  across  the  different  operational  scenarios.  SAS 
programs,  and  accompanying  technical  documentation,  for  replicating  the  simulation  are 
available  in  Appendix  D. 

Selection  of  MOS  and  Procedure  for  Estimating  Parameters  for  Computer-Generated  Samples 

- Selecting  an  MOS.  Although  a  simulation-based  analytic  strategy  was  selected,  steps 

were  taken  to  ensure  that  findings  from  our  simulation  approximated  those  of  the  real-world.  To 
do  this,  a  single  MOS  was  selected  to  provide  the  parameters  on  which  the  computer-generated 
samples  would  be  based.  The  MOS  selected  was  meant  to  reflect  the  typical  MOS. 

While  multiple  MOS  were  available  and  could  have  been  aggregated  to  form  the  typical 
(or  average)  MOS,  we  did  not  follow  this  approach  for  two  reasons.  First,  different  MOS  use 


■  •  4  Attrition  was  not  simulated  directly,  but  based  on.  simulated  performance  across  a  battery  of  training  tests.  Details 
on  how  this  was  done  appear  later  in  the  Method  section. 
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different  AA  composites  to  determine  eligibility  and  require  a  different  number  of  tests  to 
complete  training  with  different  passing  grades,  which  makes  simple  aggregation  difficult. 
Second,  the  sample  of  MOS  available  was  small  (4)  and  generally  reflected  a  convenience 
sample.  Therefore,  the  available  MOS  may  not  have  been  representative  of  the  full  population  of 
Army  jobs,  making  it  unlikely  that  aggregating  across  MOS  would  have  produced  the  typical 
MOS. 


When  selecting  the  MOS  to  serve  as  input  for  constructing  the  computer-generated 
samples,  care  was  taken  to  identify  the  MOS  from  those  available:  (a)  with  the  fewest  data 
quality  issues  (e.g.,  data  entry  errors);  and  (b)  whose  estimated  academic  attrition  rate  was 
comparable  to  that  typically  found  in  the  Army  (between  10%  and  20%).  Extensive  diagnostic 
and  screening  procedures  were  conducted  on  all  MOS.  After  reviewing  the  available  MOS,  we 
identified  55D  (Explosive  Ordnance  Disposal)  which  best  met  these  criteria  and,  thereby,  was 
chosen  for  estimating  the  real-world  parameters  on  which  the  computer-generated  samples 
would  be  based.  Specifically,  this  particular  MOS  was  selected  because:  (a)  there  were  a 
reasonable  number  of  cases  with  complete  training  performance  data;  (b)  its  cases  exhibited  the 
least  criterion  contamination;  and  (c)  its  observed  attrition  rate,  taking  into  account  retests,  best 
approximated  that  of  the  typical  MOS. 

Procedure  for  estimating  parameters.  Before  constructing  the  computer-generated 
samples,  we  selected  and  estimated  the  parameters  on  which  these  samples  would  be  based.  The 
following  parameters  were  selected:  (a)  mean,  variance,  and  distribution  of  AA  composites;  (b) 
mean,  variance,  and  distribution  of  test  scores  (e.g.,  training  performance);  (c)  AA  composite¬ 
training  performance  validities;  (d)  error  correlation  matrix  of  the  test  scores;  and  (e)  regression 
parameters  (e.g.,  intercept  and  slope)  from  regressing  test  scores  onto  AA  composite  scores.  As 
a  brief  overview,  the  following  steps  were  taken  in  estimating  the  parameters  on  which  the 
computer-generated  samples  would  be  based: 

1 .  The  dataset  for  the  selected  MOS  was  cleaned  and  prepared  for  analysis. 

2.  Relevant  parameters  were  computed  empirically  using  the  cleaned  dataset. 

3.  Our  procedure  (and  accompanying  SAS  program)  for  generating  synthetic  training 
performance  data  was  verified.  Specifically,  we  checked  that  the  parameters  (listed 
above)  of  the  synthetic  samples  approximated  those  of  the  selected  MOS. 

Each  of  these  steps  is  detailed  below. 

First,  after  the  MOS  was  selected,  the  dataset  containing  data  for  that  MOS  was  prepared 
for  analysis.  The  available  data  were  cleaned  and  the  integrity  checked.  More  specifically,  in 
preparing  the  dataset  we:  (a)  merged  multiple  records  for  a  single  participant  into  a  single  record; 
(b)  screened  data  for  coding  errors,  such  as  checking  to  ensure  that  the  minimum  and  maximum 
values  for  test  scores,  AA  composite  scores,  and  other  relevant  variables  were  within  the 
legitimate  range;  and  c)  deleted  problem  cases  (e.g.,  participants  having  AA  composite  scores 
below  the  operational  cut  score)  from  the  dataset,  so  as  not  to  bias  empirically  estimated 
parameters. 
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Second,  using  the  dataset  cleaned  in  the  preceding  step,  key  parameters  were  computed 
empirically  from  the  available  data.  Key  parameters  computed  included:  (a)  mean  and  variance 
of  AA  composite  scores5;  (b)  mean  and  variance  of  test  scores  (e.g.,  training  performance);  (c) 
AA  composite-training  performance  validities;  (d)  error  correlation  matrix  of  the  test  scores;  and 
(e)  regression  parameters  (e.g.,  intercept  and  slope)  from  regressing  test  scores  onto  AA 
composite  scores6.  These  parameters  became  the  real-world  data  from  which  the  computer¬ 
generated  samples  would  be  constructed.  This  is  important,  as  one  of  our  goals  was  to  simulate 
computer-generated  samples  whose  properties  (e.g.,  training  performance)  would  approximate 
those  of  operational  MOS  as  closely  as  possible.  Only  data  on  cognitive  aptitude  (as  reflected  in 
AA  composite  score)  were  incorporated  into  the  model.  Therefore,  properties  (e.g., 
demographics)  other  than  cognitive  aptitude  are  not  (directly)  reflected  in  the  synthetic  samples. 
This  procedure  was  followed  because  the  focus  of  the  current  study  is  on  changes  in  enlistment 
standards,  and  cognitive  aptitude  is  directly  tied  to  the  setting  and  application  of  those  standards. 

For  the  third  (and  final)  step,  after  computing  relevant  parameters  empirically  for  the 
selected  MOS,  we  checked  the  integrity  of  the  procedure  (and  accompanying  SAS  program)  for 
generating  the  samples  of  synthetic  data.  Specifically,  we  verified  that  the  parameters  (listed 
above)  estimated  empirically  from  the  selected  MOS  would  approximate  the  same  parameters 
obtained  from  pilot  samples  of  synthetic  data.  Technical  details  related  to  our  simulation 
procedure  are  documented  in  Appendix  D.  In  brief,  the  procedure  is  regression-based  using 
empirically  derived  intercepts  and  weights.  These  regression  parameters  were  then  used  to 
simulate  samples  of  synthetic  data  that  closely  reflect  the  properties  of  the  selected  MOS  (e.g., 
AA  composite-test  score  validities)  on  which  the  samples  are  based. 

To  verify  the  integrity  of  the  procedure  and  accompanying  SAS  program  for  producing 
the  computer-generated  samples,  we  conducted  a  series  of  pilot  simulations.  For  the  pilot 
simulations,  we  constructed  four  synthetic  samples  (j¥=1  0,000)  based  on  the  real-world 
parameters  computed  for  the  selected  MOS.  Each  sample  contained  synthetic  AA  composite  and 
training  performance  data  for  an  MOS  school.  For  each  synthetic  sample,  we  compared:  (a)  the 
simulated  AA  composite  scores  to  real-world  AA  scores;  and  (b)  the  properties  of  the  simulated 
training  performance  data  to  those  of  the  real-world  performance  data. 

First,  to  verify  the  simulated  AA  composite  scores,  we  computed  the  mean  and  variance 
of  these  scores  and  compared  them  to  those  of  real-world  AA  scores.  Additionally,  we  checked 


Since  only  a  single  AA  composite  is  used  operationally  to  set  minimum  enlistment"  standards  for  most  MOS,  data 
(e.g.,  mean,  SD,  and  AA  composite-training  performance  validities)  for  a  single  AA  composite,  specifically  the 
composite  most  relevant  to  the  MOS  in  question,  were  computed. 

6  Based  on  the  data  available,  recruits  were  required  to  pass  17  tests  to  complete  technical  training  for  MOS  55D. 
The  following  are  the  observed  regression  parameters  (in  parentheses-constants  first,  unstandardized  regression 
weights  second)  for  each  of  these  17  tests:  Testl  (63.56,  .24);  Test2  (86.83,  .06);  Test3  (62.56,  .24);  Test4  (82.93, 
.10);  Test5  (74.32,  .18);  Test6  (80.10,  .11);  Test7  (75.46,  .11);  Test8  (39.49,  .43);  Test9  (68.84,  .17);  TestlO  (63.04, 
.23);  Testl  1  (60.04,  .25);  Testl2  (68.74,  .18);  Testl3  (85.61,  .02);  Testl4  (69.99,  .22);  Testl5  (62.45,  .24);  Testl6 
(86.41,  .03);  Testl7  (79.77,  .13).  Note,  these  and  other  key  parameters  computed  from  the  MOS  55D  sample,  which 
became  the  basic  input  to  the  simulation,  can  be  found  in  Appendix  D. 
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the  distribution  (divided  into  quartiles)  of  simulated  AA  scores  to  that  of  the  real-world  scores. 
Both  within  each  sample  and  averaged  across  all  four  samples,  the  mean,  variance,  and 
distribution  of  synthetic  AA  composite  scores  closely  matched  that  of  the  real-world  AA 
composite  data. 

Second,  to  verify  the  simulated  training  performance  data,  we  did  the  following.  Using 
the  synthetic  samples,  we  computed  the  mean,  variance,  and  distribution  of  training  performance 
(or  test)  scores  and  compared  these  descriptives  to  those  of  the  real-world  parameters.  Both 
within  each  sample  and  across  all  samples,  the  mean,  variance,  and  distribution  of  synthetic  test 
scores  closely  approximated  their  corresponding  real-world  parameters.  Next,  we  calculated  the 
AA  composite-training  performance  validities.  As  with  the  basic  descriptives,  validities 
estimated  for  the  synthetic  samples  were  comparable  to  real-world  validities,  across  all  training 
tests.  We  then  estimated  the  error  correlation  matrix7  for  the  training  performance  tests  (i.e., 
correlations  among  the  residuals  for  the  different  tests).  Consistent  with  other  parameters,  the 
error  correlation  matrix  of  the  synthetic  samples  matched  the  matrix  used  to  simulate  the 
synthetic  data.  Finally,  we  regressed  simulated  training  performance  scores  onto  AA  composite 
scores  (for  each  of  the  tests  required  to  complete  training  for  the  selected  MOS).  We  then 
compared  the  regression  parameters  (i.e.,  intercepts  and  regression  weights)  computed  for  the 
synthetic  samples  to  those  estimated  using  the  real-world  data.  Within  each  sample  and  across 
all  samples,  the  regression  parameters  obtained  for  the  synthetic  samples  were  comparable  to  the 
corresponding  real-world  parameters,  across  the  full  battery  of  training  tests. 

As  an  additional  pilot,  we  varied  key  parameters  (e.g.,  mean  and  variance  of  AA 
composite  scores)  to  see  if  these  changes  would  be  reflected  in  subsequent  synthetic  samples. 
Multiple  synthetic  samples  (#=10,000)  were  generated  based  on  these  modifications.  Results 
from  this  additional  pilot  further  verified  the  integrity  of  the  procedure  and  accompanying  SAS 
program,  as  changes  in  selected  parameters  were  accurately  reflected  in  the  updated  synthetic 
samples. 

In  summary,  both  the  procedure  and  program  for  generating  synthetic  samples  for  use  in 
the  simulation  were  satisfactorily  verified.  Taken  together,  the  findings  from  the  pilot  showed 
that  our  procedure  successfully  replicated  and  approximated  real-world  data,  producing  samples 
that  closely  reflected  the  properties  of  an  actual  MOS. 


Selection  of  Operational  Scenarios  for  Assessing  Effects  of  Changes  in  Enlistment  Standards  on 
Attrition  Rates 

Toplace  the  effects  of  changing  enlistment  standards  on -attrition  rates  in  context,  we 
sought  to  identify  operational  scenarios  Army  personnel  managers  are  likely  to  encounter  on  the 


'  Note,  the  error  correlation  matrix  computed  empirically  for  the  real-world  MOS  contained  negatively  correlated 
residuals.  Prior  to  running  the  full  simulation,  we  compared  results  from  our  pilot  modeling  negatively  correlated 
residuals  versus  not  modeling  negatively  correlated  residuals,  and  there  were  no  substantial  differences  in  the 
estimates  produced.  Therefore,  to  simplify  the  simulation,  and  because  we  did  not  have  sufficient  data  to  fully 
investigate  these  issues,  negatively  correlated  residuals  and  residuals  close  to  zero  were  set  to  zero  for  purposes  of 
generating  the  synthetic  data  used  for  our  simulation. 
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job.  Doing  so  would  assist  Army  managers  in  understanding  the  implications  of  changes  in 
enlistment  standards  on  academic  attrition  across  a  variety  of  different  operational  environments. 
To  this  end,  we  identified  factors  (and  levels  to  these  factors)  that  would  meaningfully  impact  the 
interpretations  Army  managers  make  about  proposed  changes  in  enlistment  standards  and 
attrition  rates.  The  specific  factors  included  and  the  rationale  for  including  each  are  described 
below. 


School  sample  size.  Operationally,  MOS  differ  in  size;  some  MOS  are  larger  than  others. 
Therefore,  the  number  of  Soldiers  receiving  training  for  a  particular  MOS  will  also  vary.  When 
making  decisions  about  raising  (or  lowering)  enlistment  standards,  the  school  sample  size  (TV)  on 
which  attrition  rates  are  based  will  meaningfully  impact  these  decisions.  When  attrition  rates  are 
based  on  school  sample  sizes  that  are  small  (e.g.,  T\r<100),  the  error  associated  with  these 
estimates  will  be  larger.  Practically,  this  means  the  “true”  attrition  rate  could  vary  considerably, 
and  may  be  quite  different  than  the  rate  observed.  For  example,  while  the  observed  rate  may  be 
15%,  due  to  error  the  “true”  attrition  rate  could  actually  be  anywhere  from  9%  to  21%.  Because 
of  this,  Army  personnel  managers  need  to  be  careful  when  deciding  whether  to  change  current 
enlistment  standards  and  how  big  of  a  change  is  necessary  to  achieve  operational  objectives.  To 
illustrate  the  potential  impact  of  error  (associated  with  school  size)  on  attrition  rate  estimates  and 
the  decisions  based  on  these  estimates,  we  varied  school  sample  size  (TV=100,  200,  400,  800, 
1600,  3200)  when  estimating  academic  attrition.  The  sample  sizes  selected  represented  levels  at 
which  we  expect  substantial,  and  practically  meaningful,  differences  in  the  amount  of  error 
associated  with  estimated  attrition  rates. 

Range  of  minimum  enlistment  standards  or  cut  scores.  Because  of  differences  in  the 
criticality  of  the  job  or  accession  goals,  minimum  enlistment  standards  will  vary  across  MOS.  In 
general,  as  enlistment  standards  are  raised  (lowered),  attrition  rates  are  expected  to  decline 
(increase).  The  attrition  rates  may  also  vary  with  enlistment  standards,  such  that  the  rate  by 
which  attrition  decreases  will  be  steeper  when  standards  are  in  the  low  to  medium  range,  but 
gradually  flattens  out  when  standards  are  high.  Attrition  rates  were  estimated  across  a  range  of 
minimum  enlistment  standards  (or  cut  scores)  to  provide  Army  personnel  managers  with 
information  about  these  effects.  Specifically,  we  estimated  attrition  rates  for  cut  scores  ranging 
from  80  to  120.  This  particular  range  of  cut  scores  was  selected  for  two  reasons.  First,  the  range 
selected  is  comparable  to  the  range  of  cut  scores  currently  employed  by  the  Army  operationally 
(84  to  1 10).  Second,  Army  accession  policy  limits  the  percentage  of  accessions  falling  below 
AFQT  category  IIIB  who  are  eligible  for  military  service;  presently,  around  2%.  Because  of  this, 
setting  the  minimum  cut  score  at  80  ensures  that  information  pertinent  to  all  enlisted  Soldiers  in 
AFQT  categories  IIIB  and  above  is  represented  in  the  simulation.  Doing  so  accurately  reflects 
the  Army’s  current  operational  environment,  where  the  overwhelming  majority  (98%)  of  its 
Soldiers  fall  into  these  categories. 

Type  of  MOS.  MOS  meaningfully  differ  in  the  nature  and  complexity  of  the  tasks 
required  for  effective  performance.  Because  of  this,  academic  attrition  rates  will  vary  by  MOS, 
such  that  attrition  rates  will  tend  to  be  higher  (or  lower)  for  some  MOS  than  others.  To  show  the 
effects  of  changes  in  enlistment  standards  on  academic  attrition  across  the  full  range  of  Army 
MOS,  we  estimated  attrition  rates  by  varying  key  parameters  that  reflected  operational 
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differences  in  MOS,  Specifically,  we  varied  parameters  reflecting  differences  in  training 
difficulty  and  job  complexity.8  A  brief  description  of  each  appears  below. 

1)  Training  difficulty.  Operationally,  training  for  some  MOS  is  more  difficult  than  others 
because  of  the  standards,  length  or  content  of  its  training  program.  To  reflect 
substantive  differences  in  training  difficulty,  we  varied  a  regression  parameter  (the 
intercept)  for  predicting  test  scores.  Higher  intercepts  reflect  less  difficult  MOS, 
whereas  lower  intercepts  reflect  more  difficult  MOS.  Conceptually,  the  level  of  the 
intercept  represents  how  well  people  would  be  expected  to  perform  independent  of 
cognitive  aptitude.  In  other  words,  it  reflects  average  training  performance  as  a 
function  of  training  difficulty.  Training  difficulty  is  expected  to  be  positively  related 
to  attrition  rates,  such  that  higher  levels  of  difficulty  produce  higher  attrition  rates. 

2)  Job  complexity.  Like  civilian  jobs,  the  occupational  requirements  of  MOS  vary  in 
their  complexity.  To  reflect  substantive  differences  in  job  complexity,  we  varied  AA 
composite-training  performance  validities,  as  performance  validities  will  vary  by 
MOS  (e.g,,  Scholarios  et  al.,  1994;  Statman,  1993).  These  differences  reflect 
meaningful  differences  in  the  content  of  the  job  (Guion,  1998;  Schmitt  &  Chan,  1998; 
Zeidner,  Johnson,  &  Scholarios,  1997).  For  example,  past  research  has  shown  that 
lower  validities  between  aptitude  tests  and  training  performance  are  associated  with 
jobs  low  in  cognitive  complexity,  whereas  higher  validities  are  associated  with  more 
cognitively  demanding  jobs  (Hunter  &  Hunter,  1984).  Job  complexity  is  expected  to 
contribute  to  attrition  rates,  such  that  jobs  high  in  cognitive  complexity  produce  lower 
attrition  rates,  while  jobs  low  in  cognitive  complexity  disproportionately  produce 
higher  attrition  rates.  Conceptually,  this  comes  about  because  applicants  scoring 
higher  on  an  aptitude  test  are  more  likely  to  be  selected  into  or  attracted  to  more 
complex  jobs,  owing  to  larger  aptitude-performance  validities.  As  cognitive  aptitude 
is  positively  related  to  training  performance  (Carretta  &  Ree,  2000;  Earles  &  Ree, 
1992;  Hunter  &  Hunter,  1984;  Olea  &  Ree,  1994;  Ree,  Carretta,  &  Teachout,  1995; 
Ree  &  Earles,  1991;  Schmidt  &  Hunter,  1998),  more  cognitively  complex  jobs,  by 
having  a  larger  proportion  of  high  aptitude  trainees,  should  display  lower  overall 
levels  of  academic  attrition. 

Two  points  should  be  emphasized  regarding  our  handling  of  job  complexity  in  the 
simulation.  First,  for  our  simulation,  we  did  not  model  the  aforementioned  selection  / 
classification  process.  Therefore,  for  purposes  of  our  simulation  the  relationship 
-  between  aptitude-performance -validities  and  . attrition  is  direct  and  represents  a 
mathematical  construction:  This  means  that  our  simulation  results  do  not  reflect  the 


s  In  varying  these  parameters,  we  kept  other  parameters  (e.g.,  regression  weights)  fixed.  An  advantage  to  this 
approach  is  that  it  simplifies  estimation  and  interpretation.  A  disadvantage  is  that  operationally,  the  fixed  parameters 
will  change  as  a  function  of  differences  in  the  parameters  we  varied.  Therefore,  readers  should  focus  on  the  pattern 
of  estimated  attrition  rates  as  a  function  of  the  variables  selected,  and  not  the  levels  of  the  attrition  rates. 
Additionally,  while  the  number  of  tests  required  to  complete  training  will  vary  across  MOS,  this  parameter  was  kept 
fixed  to  the  number  of  tests  required  for  the  particular  MOS  on  which  the  computer-generated  samples  were  based. 
This  was  done  to  simplify  inteipretation  of  results  related  to  training  difficulty  and  job  complexity. 
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actual  aptitude  distribution  of  trainees  within  an  MOS,  which  is  technically  fixed 
within  the  simulation.  Nevertheless,  the  function  of  that  relationship  in  our 
simulation  is  entirely  consistent  with  the  underlying  selection  /  classification 
mechanism  not  modeled.  Second,  readers  should  note  that  our  results  for  job 
complexity  could  be  interpreted  in  one  of  two  ways.  They  can  be  interpreted  as 
reflecting  differences  in  attrition  rates  for  MOS  that  vary  in  levels  of  job  complexity, 
as  complexity  is  meaningful  associated  with  differences  in  aptitude-performance 
validities,  which  is  our  primary  intention.  Alternatively,  results  can  be  interpreted  as 
reflecting  operational  differences  in  aptitude-training  performance  validities  (e.g., 
differences  in  validities  given  a  fixed  MOS).  Either  interpretation  is  technically 
correct.  As  the  primary  aim  of  the  study  is  to  inform  policy,  however,  we  will 
emphasize  the  first  interpretation  in  discussing  the  results  and  their  implications. 

In  summary,  the  simulation  conducted  reflected  a  mixed  design,  with  school  size  (#)  and 
minimum  enlistment  standards  as  within-group  (or  repeated)  factors,  and  training  difficulty  and 
job  complexity  as  between-group  factors.  This  information  is  summarized  in  Table  1. 


Table  1 

Design  of  Simulation 


Training  Difficulty 

Job  Complexity 

Low 

Medium 

High 

Low 

•  #(100,  200,  400, 

800,  1600,  3200) 

•  Cut  Score  (80-120) 

•  #(100,  200,  400,  800, 
1600,  3200) 

•  Cut  Score  (80-120) 

•  #(100,200,400,  800, 
1600,  3200) 

•  Cut  Score  (80-120) 

Medium 

•  #(100,200,400, 

800,  1600,  3200) 

•  Cut  Score  (80-120) 

•  #(100,  200,  400,  800, 
1600,  3200) 

•  Cut  Score  (80-120) 

•  #(100,200,400,  800, 
1600,  3200) 

•  Cut  Score  (80-120) 

High 

•  #(100,200,400, 

800,  1600,  3200) 

•  Cut  Score  (80-120) 

•  #(100,200,400,  800, 
1600,  3200) 

•  Cut  Score  (80-120) 

•  #(100,  200,  400,  800, 
1600,  3200) 

•  Cut  Score  (80-120) 

Procedure  for  Simulating  Attrition  Rates  under  Different  Operational  Scenarios 

"After  identifying  the  operational  scenarios  of  interest  to  Army  personnel  managers,  we 
simulated  academic  attrition  rates  under  these  different  scenarios.  This  section  is  organized  as 
follows.  First,  we  briefly  describe  how  attrition  data  for  the  computer-generated  samples  were 
constructed  from  synthetic  training  performance  data.  Second,  we  briefly  outline  the  procedure 
for  simulating  attrition  rates  under  the  different  operational  scenarios  described  in  the  previous 
section. 


Constructing  Attrition  Data  from  Synthetic  Training  Performance  Scores.  We  created  a 
dichotomous  outcome  variable  reflecting  academic  attrition  of  simulated  trainees  based  on  their 
synthetic  training  performance  scores.  The  values  for  this  variable  were  assigned  as  follows.  If 
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a  simulated  trainee  did  not  score  above  the  specified  minimum  passing  score9  (after  two  * 
attempts)  for  one  or  more  of  the  training  tests,  he  or  she  was  coded  as  an  academic  attrition 
(“1”);  if  all  tests  were  passed,  the  trainee  was  coded  as  not  attriting  (“0”).  Because  information 
on  retests  could  not  reliably  be  reconstructed  from  existing  data,  we  simulated  trainee  retest 
behavior  by  assuming  that  the  two  test  attempts  were  statistically  independent.  Given  that 
retraining  of  applicants  who  failed  the  first  attempt  is  likely  designed  to  give  them  a  better 
chance  of  passing  the  second  attempt,  the  assumption  of  independence  between  two' attempts  is 
expected  to  lead  to  slightly  more  conservative  (i.e.,  higher)  attrition  estimates. 

Prior  to  running  the  full  simulation,-  we  calculated  overall  attrition  rates  for  the  synthetic 
samples  generated  during  the  pilot  simulation  by  counting  the  number  of  simulated  trainees  who 
failed  to  pass  a  test  (after  a  second  attempt)  divided  by  the  total  number  of  trainees  (e.g,,  total 
sample  size).  We  also  computed  these  rates  for  a  simulation  modeling  a  single  attempt.  These 
simulated  attrition  rates  were  then  compared  to  the  observed  attrition  rate  from  our  real-world 
data. 


The  simulated  attrition  rates  from  modeling  two  attempts  ranged  from  12-13%.  These 
rates  were  comparable  to  the  real  world  attrition  rate  of  roughly  12%.  Interestingly,  there  was  a 
substantial  difference  in  the  simulated  versus  real  world  attrition  rates  when  simulated  trainees 
were  only  allowed  a  single  attempt  to  pass  each  test  (e.g.,  no  retests)  versus  two  attempts. 
Modeling  the  two  attempts  resulted  in  an  attrition  rate  significantly  closer  (12-13%  versus  40- 
45%  when  modeling  a  single  attempt)  to  the  actual,  real  world  rate.  Additionally,  the  simulated 
rates  from  modeling  two  attempts  were  somewhat  higher  (about  1%)  than  the  actual  attrition 
rate.  This  is  consistent  with  our  expectation  that  assuming  independence  would  produce  more 
conservative  (higher)  estimates,  although  materially  the  difference  does  not  appear  substantial. 
Most  likely,  this  is  due  to  the  fact  that  the  percentage  of  trainees  requiring  more  than  a  single 
attempt  for  at  least  one  test  is  generally  small.  Taken  together,  these  findings  lend  confidence  to 
the  reliability  of  our  procedure  for  simulating  retest  behavior,  assuming  independence  across  re¬ 
tests.  -  ■  -  ;  :  •-  =  -  ;•  V . >  - 


Simulating  Attrition  Rates  under  Different  Operational  Scenarios.  As  discussed  earlier,  a 
major  advantage  of  our  simulation-based  approach  is  the  ability  to  assess  error  in  estimated 
attrition  rates  and  to  demonstrate  its  impact  on  the  operational  decisions  Army  personnel 
managers  make  about  where  to  set  minimum  enlistment  standards.  This  was  accomplished  by 
generating  multiple  synthetic  samples  at  varying  sample  sizes  based  on  parameters  that  reflected 
different  operational  conditions  Army  managers  are  likely  to  encounter  in  the  field.  Simply  put, 
while  the  synthetic  samples  reflect  the  properties  of  the  MOS  used  to  generate  them,  they  are  not 
exact  replicas  of  that  MOS.  The  parameters  used  to  generate  the  synthetic- samples  represent 
population-level  information,  whereas  each  synthetic  sample  reflects  the  characteristics  of  a 
particular  sample  randomly  drawn  from  that  population.  Because  error  is  built  into  these 
estimations,  there  is  sample-to-sample  variability  in  the  properties  of  the  synthetic  samples. 
Therefore,  the  samples  behave  in  much  the  same  way  that  randomly  selected  samples  would 


9  All  tests  were  fixed  at  the  same  passing  score.  This  was  done  for  two  reasons:  first,  to  simplify  estimation;  and 
second,  because  this  information  could  not  be  reliably  obtained  from  available  data. 
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behave  in  the  real  world.  The  larger  its  sample  size,  the  more  a  synthetic  sample  will 
approximate  the  properties  of  the  corresponding  population  (e.g.,  the  parameters  on  which  the 
sample  was  based).  The  smaller  its  sample  size,  the  less  a  synthetic  sample  will  approximate  the 
properties  of  the  corresponding  population.  By  generating  multiple  synthetic  samples,  we  were 
able  to  model  the  adverse  effects  of  error  on  estimated  attrition  rates,  so  as  to  provide  practical 
recommendations  on  these  effects  when  making  critical  operational  decisions. 

As  reflected  in  Table  1  above,  there  were  a  total  of  nine  conditions  (e.g.,  Low  Difficulty, 
Low  Complexity)  in  our  simulation.  In  estimating  academic  attrition  rates,  the  same  levels  of 
sample  size  and  range  of  cut  scores  were  used  for  all  conditions  (see  Table  1).  For  technical 
documentation  and  accompanying  SAS  programs,  see  Appendix  D.  In  brief,  for  each  condition, 
we  conducted  the  following  activities: 

1 .  Using  the  parameters  relevant  to  a  given  condition,  we  replicated  2,000  samples  at  6 
different  sample  sizes  (e.g.,  100,  200, 400,  800, 1600,  and  3200);  each  sample 
containing  synthetic  training  performance  and  attrition  data.  This  means  that  for  each 
condition,  we  simulated  a  total  of  12,000  samples  (6  x  2,000  =  12,000).  The 
properties  of  these  samples  approximated  the  parameters  specified,  which  were  meant 
to  reflect  the  operational  conditions  Army  personnel  managers  were  likely  to  see  on 
the  job.  Prior  to  the  simulation,  the  integrity  of  the  procedure  and  SAS  program  for 
generating  the  synthetic  samples  had  been  successfully  verified  (see  pages  13  -  14). 

2.  For  each  of  the  2,000  computer-generated  samples,  we  estimated  the  recommended 
logistic  regression  model,  by  regressing  attrition  (a  dichotomous  dependent  variable) 
onto  synthetic  AA  composite  scores.  These  regression  parameters  became  the  input 
for  estimating  attrition  rates. 

3.  For  each  of  the  2,000  replications,  we  estimated  academic  attrition  rates,  using  the 
- regression  parameters  computed  (from  Step  2),  for  a  range  of  cut  scores.  This  was 

done  by  sample  size  ( N ).  As  discussed  previously,  the  differing  sample  sizes  ( N) 
reflected  operational  differences  in  school  size.  The  range  of  cut  scores  reflected 
operational  differences  in  minimum  enlistment  standards. 

4.  We  then  averaged  the  estimated  attrition  rates  across  all  2,000  replications  for  the  full 
range  of  cut  scores  by  N.  In  the  process  of  averaging  attrition  rates,  we  estimated  the 
corresponding  standard  errors  (SEs).  When  done,  average  attrition  rates  and 
corresponding  SEs  were  compiled  and  outputted.  For  each  condition,  the  final  output 

...  ..  consisted  of  a  single  table  summarizing  average  attrition  rates  (and. standard  errors) 
by  AA  composite  cut  score  and  N  (see  Appendix  B).  The  composite  utilized  in  the 
simulations  is  General  Maintenance  (GM). 
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RESULTS 


Results  are  organized  by  the  factors  (school  size,  range  of  enlistment  standards,  training 
difficulty,  job  complexity)  we  varied  to  reflect  the  different  operational  scenarios  Army 
personnel  managers  were  likely  to  encounter  on  the  job.  Tables  and  Figures  referenced  in  this 
section  can  be  found  in  Appendices  B  and  G.  Readers  should  note  that  the  attrition  rates  reported 
are  estimates  based  on  simulated  data  and  are  not  associated  with  a  particular  MOS.  Because  of 
this,  the  rates  reported  should  not  be  used  to  make  operational  decisions  for  a  given  MOS,  as 
they  do  not  represent  actual  MOS  school  attrition  rates.  For  operational  usage,  readers  are 
advised  to  apply  the  proposed  methodology  of  the  current  study  to  available  real-world  data  for 
the  MOS  of  interest.  Recommendations  for  doing  so  are  detailed  in  the  Discussion  section.  The 
SAS  program  (with  documentation)  for  running  an  attrition  rates  analysis  using  actual  MOS 
school  data  appears  in  Appendix  E. 


School  Sample  Size  (N) 

School  sample  size  (N)  reflects  operational  differences  in  the  number  of  Soldiers  trained 
for  a  selected  MOS.  We  varied  school  sample  size  (A=100;  200;  400;  800;  1600;  3200)  to 
represent  levels  at  which  we  expected  to  see  substantial,  and  practically  meaningful,  differences 
in  the  amount  of  error  associated  with  estimated  attrition  rates. 

As  evident  from  Tables  1  to  9  (see  Appendix  B)10,  attrition  rates  do  not  vary  significantly 
as  a  function  of  N.  For  any  given  cut  score,  the  estimated  attrition  rate  is  generally  the  same 
irrespective  of  N.  This  trend  is  most  apparent  when  looking  at  Figures  1  to  9  (see  Appendix  C). 
For  example,  when  viewing  Figure  1,  one  sees  that  across  the  full  range  of  cut  scores,  the 
estimated  attrition  rates  for  the  varying  levels  of  N  substantially  overlap,  such  that  there  is 
basically  a  single  line  showing  the  trend  in  attrition  rates  by  cut  score.  Because  of  the  amount  of 
overlap,  the  lines  reflecting  the  different  levels  of  N  cannot  be  meaningfully  differentiated. 

There  are  small  differences  (about  .01)  in  attrition  rates  by .N  for  some  of  the  lower  cut  scores, 
but  overall,  attrition  rates  do  not  vary  significantly  as  a  function  of  N.  These  findings  are  not 
surprising  given  the  large  number  of  synthetic  samples  (2,000)  generated  for  the  simulation. 
Because  of  the  large  number  of  synthetic  samples,  the  average  attrition  rate  reported  in  Tables  1 
to  9  should  approximate  that  of  the  corresponding  population  (or  “true”)  attrition  rate. 

The  above  findings  should  not  be  interpreted  to  mean  that  N  will  not  impact  the  observed 
attrition  rate  estimated  from  a  sample  of  MOS  school  data,  as  would  be  the  case  when  this 
analysis  is  performed  operationally.  .  Rather,  these  findings  suggest  that  estimated  attrition  rates 
are  not  expected  to  be  systematically  biased  in  one  direction  or  another  (e.g.,  estimated  attrition 
rates  will  always  be  higher  or  lower  than  the  “true”  attrition  rate).  The  impact  of  N  on  school 


10  Please  note  that  when  the  cut  score  was  high  for  some  of  the  conditions  (e.g.,  low  difficulty,  high  complexity)  so 
that  the  estimated  attrition  rate  is  close  to  zero,  several  of  the  2,000  replications  for  the  smaller  sample  sizes  (AM  00 
or  200)  were  not  included  in  the  estimation  of  the  average  attrition  rate  (and  SE)  because  there  was  no  attrition  on 
which  to  base  estimates.  Because  the  percentage  of  replications  dropped  (Out  of  the  total)  was  generally  small  and 
zero  attrition  is  practically  not  going  to  be  a  concern  to  Army  managers,  we  did  not  correct  this. 
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attrition  rates  (estimated  from  sample  data)  is  best  examined  with  the  standard  error  (SE),  which 
measures  the  variability  of  these  estimates  from  sample  to  sample.  Operationally,  the  amount  of 
SE  matters  because  it  speaks  to  the  precision  of  estimated  attrition  rates.  For  example,  if  the 
observed  attrition  rate  is  .13  (or  13%)  and  the  SE  is  .04,  then  the  “true”  attrition  rate  could  be 
anywhere  from  .05  (5%)  to  .21  (21%)  within  generally  accepted  levels  of  statistical  confidence 
(about  95%).  Clearly,  the  decision  made  about  where  to  set  enlistment  standards  would  vary 
considerably  depending  on  which  of  these  values  (l.e.,  5%  vs.  13%  vs.  21%)  most  accurately 
reflects  the  “true”  attrition  rate. 

As  can  be  seen  from  Tables  1  to  9,  it  is  apparent  that  as  N  increases,  the  precision  (as 
reflected  in  the  SEs)  of  the  estimates  increases,  as  expected.  More  specifically,  looking  at 
Figures  10  to  18,  one  sees  two  notable  trends  regarding  the  behavior  of  SE.  First,  SE  is  inversely 
related  to  N,  such  that  as  N  increases,  SE  decreases.  From  N  =  100  to  200,  SE  is  large  and  the 
reduction  in  SE  (from  N= TOO  to  200)  is  modest,  roughly  26  -  35%  (see  Figures  19  -  27). 

Moving  to  N=  400  further  reduces  SE  (about  48  -  55%  from  N =  100),  but  the  resulting  SE  is 
still  large  relative  to  the  attrition  rate.  Starting  at  N  =  800,  we  start  to  see  a  meaningful  reduction 
in  SE  (about  62  -  69%),  that  progressively  levels  off  as  N  doubles  from  800  to  1600  (74  -  79%) 
and  then  again  to  3200  (8 1  -  83%). 

The  second  trend  evident  from  Figures  10  to  18,  is  the  relationship  between  SE  and  cut 
score.  Specifically,  as  the  cut  score  is  raised,  SE  decreases.  However,  this  decrease  in  SE  is 
largely  a  consequence  of  the  corresponding  decrease  in  the  magnitude  of  attrition  rates.  That  is, 
smaller  absolute  standard  errors  are  expected  when  estimating  the  lower  attrition  rates  associated 
with  higher  cut  scores.  This  is  more  visible  when  looking  at  Figures  28  to  36.  As  can  be  seen 
from  these  figures,  the  amount  of  SE  relative  to  the  magnitude  of  the  attrition  rate  (reflected  in 
the  coefficient  of  variation)  appears  to  be  uniform  across  the  cut  score  range  except  at  the 
highest,  most  extreme  score  in  the  range  (where  AA  =  120). 

In  summary,  school  size  significantly  impacts  the  level  and  accuracy  of  attrition  rates 
estimated  from  sample  school  data.  Most  importantly,  these  estimates  are  likely  to  be  less  biased 
and  more  precise  with  larger  N.  This  relationship  can  have  major  implications  for  the 
operational  decisions  Army  personnel  managers  must  make  regarding  where  to  set  minimum 
enlistment  standards. 


Range  of  Minimum  Enlistment  Standards 

The  range  of  minimum  enlistment  standards  reflects  operational  differences  in  the 
criticality  of  the  job  or  accession  goals  associated  with  a  targeted  MOS.  We  simulated  attrition 
rates  across  a  range  of  cut  scores  (80  to  120)  Army  personnel  managers  are  likely  to  use 
operationally  when  setting  enlistment  standards. 

As  expected,  the  raising  (or  lowering)  of  cut  scores  is  meaningfully  associated  with 
academic  attrition,  such  that  attrition  rates  are  higher  when  cut  scores  are  lower.  Referring  back 
to  Tables  1  to  9,  and  Figures  1  to  9,  and  ignoring  other  factors  (e.g.,  training  difficulty,  job 
complexity),  one  sees  that  attrition  rates  steadily  decline  as  the  cut  score  increases.  This  trend 
holds  for  all  levels  of  N.  For  example,  looking  at  Figure  5,  the  attrition  rate  (with  corresponding 
cut  score  in  parentheses)  drops  from  .30  (80)  to  .15  (100)  to  .05  (120).  Likewise,  as  the  cut  score 
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increases,  the  SE  decreases  (see  Figures  10  to  18);  although,  as  reported  in  the  preceding  section, 
the  size  of  SE  relative  to  the  underlying  attrition  rate  generally  appears  to  be  uniform. 

In  summary,  estimated  attrition  rates  are  related  to  cut  scores,  such  that  higher  cut  scores 
are  associated  with  lower  attrition  rates.  However,  the  relative  precision  of  these  estimated 
attrition  rates  appears  constant  across  cut  score.  At  an  operational  level,  this  means  that  the 
accuracy  of  sample-based  attrition  rates  is  expected  to  be  about  the  same  irrespective  of  cut 
score. 


Training  Difficulty 

Operationally,  the  training  difficulty  of  MOS  will  vary  because  of  differences  in  the 
standards,  length,  or  content  of  training.  We  varied  a  regression  parameter  (the  intercept)  for 
predicting  test  scores  to  reflect  low,  medium,  and  high  training  difficulty.  This  enabled  us  to 
project  academic  attrition  across  a  range  of  MOS  that  varied  in  training  difficulty,  just  as  MOS 
would  in  the  field. 

As  expected,  academic  attrition  meaningfully  varies  as  a  function  of  training  difficulty. 
As  can  be  seen  in  Tables  1  to  9  and  Figures  37  to  40,  and  ignoring  other  factors  (e.g.,  job 
complexity),  attrition  rates  are  higher  when  training  difficulty  is  high,  and  lower  when  difficulty 
is  low.  For  example,  looking  at  Figures  37  to  40,  one  sees  that  attrition  rates  are  consistently 
related  to  training  difficulty,  irrespective  of  A  and  cut  score,  such  that  greater  training  difficulty 
is  associated  with  higher  attrition  rates.  However,  the  differences  in  attrition  rates  by  training 
difficulty  progressively  get  smaller  as  the  cut  score  increases.  For  example,  where  A  =  400  (see 
Figure  38),  estimated  attrition  rates  differ,  on  average,  about  .15  (15%)  for  a  cut  score  of  80, 
about  .12  (12%)  for  a  cut  score  of  100,  and  .05  (5%)  for  a  cut  score  of  120. 

There  are  two  trends  to  note  related  to  the  estimated  SE  (precision)  of  observed  attrition 
rates  (see  Figures  41  to  43).  First,  SE  varies  by  A,  irrespective  of  cut  score.  Consistent  with 
earlier  findings,  and  as  expected,  the  SE  drops  as  A  increases.  This  trend  holds  for  all  levels  of 
training  difficulty.  The  second  trend  of  note  is  that  the  SE,  on  average,  is  smaller  when  training 
difficulty  is  high,  except  for  the  upper  cut  score  range  (e.g.,  100  -  105).  Likewise,  except  for  the 
lower  cut  score  range  (e.g.,  80  -  85),  SE  is  higher,  on  average,  when  training  difficulty  is  low. 
This  is  unusual,  as  we  expect  SE  to  reflect  differences  in  the  magnitude  of  the  attrition  rates 
between  difficulty  levels  (e.g.,  lower  attrition  rates  should  be  associated  with  lower  SE,  and 
higher  rates  with  higher  SE). 

In  summary,  consistent  with  our  predictions,  attrition  rates  are  substantively  related  to 
training  difficulty,  such  that  the  more  difficult  the  training  the  lower  the  attrition.  The  relative 
precision  of  the  attrition  rate  estimates  (reflected  by  SE)  appears  to  be  relatively  equivalent 
across  difficulty  level,  particularly  at  higher  levels  of  A.  Practically,  this  means  that  sample- 
based  attrition  rates  are  expected  to  exhibit  similar  levels  of  imprecision  irrespective  of  the 
difficulty  level  of  the  targeted  MOS.  We  consider  the  practical  implications  of  these  findings 
further  in  the  Discussion  section. 


22 


Job  Complexity 

Like  civilian  occupations,  Army  MOS  vary  in  their  complexity.  To  reflect  substantive 
differences  in  job  complexity,  we  varied  AA  composite-training  performance  validities  to  reflect 
meaningful  differences  in  the  content  of  the  job.  Specifically,  past  research  has  shown  that  lower 
validities  between  aptitude  tests  and  training  performance  are  associated  with  jobs  low  in 
cognitive  complexity*  whereas  higher  validities  are  associated  with  more  cognitively  demanding 
jobs  (Hunter  &  Hunter,  1984).  As  with  training  difficulty  (see  above),  varying  performance 
validities  enabled  us  to  simulate  attrition  rates  for  the  full  range  of  MOS  Army  personnel 
managers  will  encounter  operationally. 

As  predicted,  academic  attrition  is  meaningfully  related  to  job  complexity,  although  there 
are  some  patterns  related  to  SE  that  we  thought  were  unusual.  As  can  be  seen  in  Tables  1  to  9 
and  Figures  44  to  47,  and  ignoring  other  factors  (e.g.,  training  difficulty),  attrition  rates  are 
generally  higher  when  cognitive  complexity  is  low,  and  lower  when  cognitive  complexity  is 
high.  As  a  specific  example,  looking  at  Figures  44  to  47,  one  sees  that  attrition  rates  are 
meaningfully  associated  with  cognitive  complexity,  irrespective  of  N  and  cut  score,  such  that 
greater  cognitive  complexity  tends  to  correspond  to  lower  attrition  rates.  Unlike  training 
difficulty,  the  relative  drop  in  attrition  rates  across  the  cut  score  range  is  substantially  greater 
when  job  complexity  is  higher.  For  example,  in  the  middle  cut  score  range  from  90  through  110, 
the  17-point  drop  (from  49%  to  32%)  represents  a  35%  relative  decrease  in  attrition.  In  contrast, 
the  14-point  drop  (from  23%  to  9%)  and  16-point  drop  (from  18%  to  2%)  for  medium  and  high 
job  complexity,  while  not  too  different  in  comparison  to  the  17-point  drop  for  low  complexity, 
represent,  respectively,  61%  and  89%  decrease  in  attrition.  These  observations  are  consistent 
with  the  higher  discrimination  power  possible  with  higher  validities. 

Comparable  to  the  findings  for  training  difficulty,  there  are  two  trends  worth  noting  with 
respect  to  the  estimated  SE  (precision)  of  observed  attrition  rates  (see  Figures  48  to  50).  First,  as 
expected,  the  SE  decreases  as  N  increases  irrespective  of  cut  score.  This  trend  holds  for  all 
levels  of  cognitive  complexity.  A  second  trend  that  was  quite  unusual  is  that  the  SE  is  greater 
for  high  complexity  compared  to  low  complexity  jobs  in  the  lower  cut  score  range  (e.g.,  80  -  85), 
but  is  lower  in  the  higher  cut  score  range  (e.g.,  100  -  105).  This  trend  holds  even  though  the 
attrition  rate  for  high  complexity  jobs  is  consistently  lower  compared  to  low  complexity  jobs. 

This  is  unusual  in  that  we  typically  associate  lower  SE  with  the  lower  magnitudes  of  attrition 
rate. 


In  summary,  there  is  a  significant  relationship  between  job  complexity  and  attrition  rates, 
such  that  (generally)  the  more  complex  the  job  the  lower  the  proportion  of  Soldiers  failing  to 
successfully  complete  training  requirements.  Equally  important  is  that  the  relative  drop  in 
attrition  associated  with  increasing  the  cut  score  is  substantially  greater  for  higher  validities  or 
job  complexity.  Please  note  that  operationally  all  these  findings  can  also  be  interpreted  as 
applying  to  differences  in  AA  composite-training  performance  validities  (low,  medium,  and 
high),  as  validities  will  likely  vary  across  MOS  within  the  same  level  of  complexity. 
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DISCUSSION 


To  facilitate  our  discussion,  this  section  is  organized  into  three  main  parts.  First,  we 
place  the  results  of  our  simulation  in  context,  by  integrating  our  findings  on  the  effects  of  raising 
(or  lowering)  enlistment  standards  on  academic  attrition.  In  particular,  we  consider  the  practical 
implications  of  these  findings.  Second,  we  provide  concrete  recommendations  for  conducting 
analyses  of  attrition  rates  operationally.  Specifically,  we  make  recommendations  regarding:  (a) 
the  collection  and  identification  of  requisite  data  for  analysis;  (b)  planning  an  attrition  rates 
analysis;  and  (c)  interpretation  of  the  effects  of  adjusted  cut  scores  on  attrition.  These 
recommendations  are  meant  to  serve  as  a  guide  to  Army  researchers  and  personnel  managers 
responsible  for  performing  attrition  rates  analysis  for  decision-making  purposes.  Finally,  we 
conclude  our  discussion  by  making  suggestions  for  future  research. 

Effects  of  Changing  Minimum  Enlistment  Standards  on  Academic  Attrition 

Results  from  our  simulation  showed  that,  consistent  with  previous  research  (e.g.,  Carretta 
&  Ree,  2000;  Earles  &  Ree,  1992;  Hunter  &  Hunter,  1984;  Olea  &  Ree,  1994;  Ree  et  al.,  1995; 
Ree  &  Earles,  1991;  Schmidt  &  Hunter,  1998),  cognitive  aptitude  is  a  good  predictor  of  training 
performance.  This  finding  extends  recent  Army  research  on  the  validity  of  the  new  AA 
composites  (e.g.,  Zeidner  et  al,  2000)  by  using  an  updated  database  and  a  criterion  specific  to 
training  success.  Therefore,  knowledge  of  enlisted  Soldiers’  cognitive  aptitude,  reflected 
operationally  in  AA  composite  scores,  is  a  useful  tool  for  informing  training  decisions. 
Practically,  this  means  that  the  setting  of  minimum  enlistment  standards  can  (and  should)  impact 
the  level  of  academic  attrition.  Consistent  with  this,  our  results  confirmed  that,  in  general, 
raising  enlistment  standards  decreases  academic  attrition,  while  lowering  these  standards 
increases  attrition.  However,  when  interpreting  this  tendency,  decision-makers  should  be 
cautious,  as  it  depends  on  several  factors.  These  factors  could  significantly  impact  how  the 
effects  of  raising  (or  lowering)  enlistment  standards  on  attrition  rates  are  interpreted  and, 
thereby,  what  decision  is  best  for  achieving  a  desired  level  of  attrition.  These  factors  and  their 
practical  implications  are  as  follows. 

First,  the  level  and  rate  of  change  in  attrition  is  dependent  on  the  MOS.  Specifically, 
MOS  vary  in  training  difficulty  and  job  complexity.  (Readers  are  reminded  that  statements  made 
about  job  complexity  apply  equally  to  AA  composite-training  performance  validities.)  These 
characteristics  impact  academic  attrition.  For  example,  attrition  rates  will  be  higher  for  MOS 
whose  training  is  more  difficult  (apart  from  cognitive  requirements);  and  attrition  rates  will 
likely  be  higher  for  MOS  whose  job  requirements  are  less  cognitively  demanding  (as  discussed 
earlier).  This  means  that  for  some  MOS  reducing  academic  attrition,  by  way  of  enlistment 
standards,  may  be  operationally  difficult.  For  these  MOS  achieving  targeted  levels  of  attrition 
rates  may  be  infeasible,  as  they  would  require  a  substantial  raise  (or  drop)  in  enlistment  standards 
that  would  not  be  feasible  for  operational  reasons  (e.g.,  accession  goals).  For  example,  for  MOS 
where  training  difficulty  is  high,  reducing  attrition  rates  by  50%  would  necessitate  a  roughly  20 
to  25-point  adjustment  in  the  requisite  cut  score;  reductions  of  33%  (one-third)  would  require  a 
10  to  15-point  adjustment.  While  major  reductions  (e.g.,  10%  or  more)  in  attrition  rates  require 
significant  changes  in  enlistment  standards  for  all  MOS,  for  some  MOS  reaching  a  particular 
(low)  level  of  attrition  or  achieving  a  substantial  drop  in  attrition  is  especially  problematic.  One 
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potential  explanation  of  why  academic  attrition  is  higher  in  some  of  the  simulated  MOS  than 
others,  specifically  those  low  in  job  complexity,  is  that  cognitive  aptitude  appears  to  carry 
limited  utility  as  a  predictor  of  training  success.  Operationally,  this  also  leads  to  a  greater 
percentage  of  person-job  mismatching  between  trainees  and  MOS,  which  could  additionally 
contribute  to  attrition. 

In  practical  terms,  these  findings  suggest  that  reducing  or  reaching  a  desired  level  of 
academic  attrition  in  some  MOS  (e.g.,  those  with  high  training  difficulty,  those  with  low  job 
complexity)  may  be  best  achieved  by  methods  other  than  adjustments  in  minimum  enlistment 
standards,  from  which  it  follows  that  training  decisions  could  be  greatly  improved,  particularly 
for  some  MOS,  by  incorporating  information  other  than  cognitive  aptitude  into  personnel  and 
assignment  decisions.  .  •  xi.;:  . 

A  second  factor  that  substantially  impacts  the  interpretation  of  attrition  rates  is  school 
sample  size  (TV).  Operationally,  MOS  attrition  rates  observed  and  reported  to  Army  managers  for 
making  personnel  decisions,  such  as  setting  minimum  enlistment  standards,  are  sample-based 
estimates,  as  opposed  to  the  population  attrition  rates,  and  therefore,  contain  error.  For  instance, 
looking  at  Table  5 ,N=  200  and  a  cut  score  of  95,  one  sees  that  the  estimated  attrition  rate  is  .19 
(19%).  However,  treating  this  as  the  sample-based  estimate  it  is,  when  we  take  into  account 
error  (measured  by  the  standard  deviation  of  the  estimated  attrition  rate),  the  true  attrition  rate  is 
really  anywhere  from  .1 1  (1 1%)  to  .27  (27%).  This  is  extremely  important  from  an  operational 
perspective. 


To  illustrate,  consider  that  an  Army  personnel  manager’s  goal  is  to  achieve  an  academic 
attrition  rate  of  15%.  The  observed  attrition  rate,  based  on  available  sample  data  (jV—200),  is 
presently  around  19%.  On  its  face,  this  means  that  attrition  is  currently  off  by  about  4%  from  the 
designated  target.  To  reach  his/her  specified  goal  (15%),  the  manager  will  most  likely  need  to 
increase  the  cut  score,  so  as  to  reduce  attrition  by  the  desired  amount  (4%).  This  translates  into  a 
5  or  6-point  increase  in  the  current  cut  score. -However,  if  error  is  taken  into  account,  we  see  that 
the  “true”  attrition  rate  is  actually  somewhere  between  1 1%  and  27%.  This  suggests  that  attrition 
could  already  be  at  the  targeted  level  (15%),  in  which  case  an  increase  in  the  current  cut  score  is 
not  needed.  On  the  other  hand,  it  also  suggests  that  attrition  is  potentially  worse  than  it  appears 
(over  19%),  in  which  case  a  bigger  increase  in  the  cut  score  is  needed  to  reach  the  manager’s 
operational  goal  with  confidence.  The  course  of  action  the  Army  manager  should  take  depends 
on  which  of  these  scenarios  is  “correct.”  However,  it  is  difficult  to  tell  which  scenario  is  most 
“correct”,  given  the  level  of  error  (and  inaccuracy)  in  the  observed  attrition  rate.  This  is  not 
trivial,  since  raising  the  cut  score  (to  lower  attrition)  comes  with  a  potential  tradeoff  in  meeting 
accession  goals.  If  MOS  attrition  is  already  at  its  targeted  level,  then  increasing  the  cut  score 
could  adversely,  and  unnecessarily,  impact  accession  goals;  a  tradeoff,  which  in  this  case,  a 
manager  did  not  have  to  make.  More  precise  estimates  of  these  rates  would  ultimately  make  this 
decision-making  process  easier  and  would  avoid  forcing  managers  to  make  poor  decisions. 

The  point  raised  in  the  example  above  is  that  we  can  never  be  100%  sure  about  the  “true” 
attrition  rate  for  a  given  cut  score  from  sample  data.  Consequently,  what  changes  in  the  cut 
score  are  needed  to  achieve  a  targeted  attrition  rate  is  never  certain.  Fortunately,  however,  the 
level  of  uncertainty  associated  with  cut  score  changes  goes  down  as  the  sample  size  is  increased. 
To  illustrate,  we  continue  the  example  above  by  referring  to  Figures  51  and  52.  The  points  on 
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-the  curve  represent  the  “true”  attrition  rate  that  is  unknown  and  unobservable  from  sample  data. 
The  vertical  bars  represent  the  likely  range  of  attrition  rate  estimates  that  we  are  going  to 
actually  observe  using  sample  sizes  of  200  or  800.  (Note  that  the  end  points  of  the  vertical  bars 
are  two  SE  from  the  center;  approximately,  there  is  95  percent  probability  that  the  estimated 
attrition  rates  fall  within  the  interval.)  The  potential  observed  rates  represented  by  the  vertical 
bars  will  become  the  basis  of  cut  score  decisions.  Using  a  sample  size  of  200,  an  Army  manager 
will  have  to  raise  the  cut  score  to  1 15  in  order  to  be  statistically  confident  that  the  attrition  rate 
will  be  no  more  than  10%.  The  interval  from  107  to  1 14  represents  “missed  opportunities”  to  set 
a  lower  cut  score  that  can  achieve  the  10%  target;  this  is  due  to  imperfect  sample  information. 
Using  a  sample  size  of  800,  the  manager  can  set  the  cut  score  to  a  lower  value  of  1  fl  and  be 
.  equally  confident  that  the  attrition  rate  would  be  10%  or  below.  In  this  case,  the  interval  of 
missed  opportunities  is  much  shorter,  from  107  to  1 10.  In  other  words,  increasing  the  sample 
size  minimizes  the  missed  opportunities  to  adjust  the  cut  score  to  the  minimum  required  to 
achieve  a  targeted  attrition  with  confidence.  Operationally,  this  means  preserving  as  much  of  the 
eligible  applicant  pool  as  possible. 

In  summary,  the  preceding  examples  highlight  how  error  can  complicate  operational 
decisions:  (a)  about  whether  to  adjust  current  enlistment  standards;  and  (b)  how  much  to  adjust 
these  standards  to  achieve  desired  objectives. 

Results  from  our  simulation  show  that  the  amount  of  error  in  sample-based  attrition  rates 
is  strongly  related  to  school  sample  size  (i¥).  Specifically,  the  smaller  the  sample  size,  the  more 
inaccurate  (and  potentially  misleading)  is  the  estimated  attrition  rate  based  on  sample  data.  The 
bigger  the  sample  size,  the  more  accurate  (and  less  misleading)  is  the  estimated  attrition  rate.  In 
practical  terms,  our  findings  suggest  that  the  accuracy  of  sample-based  attrition  rates  is  positively 
related  to  school  sample  size  (. N ),  and  decision-making  will  be  greatly  improved  by  taking  into 
account  sample  size  when  estimating  MOS  attrition  rates. 

Recommendations  for  Analyzing  Academic  Attrition  Rates 

In  this  section,  we  detail  recommendations  for  analyzing  academic  attrition  rates 
estimated  from  MOS  school  sample  data.  Specifically,  we  make  recommendations  pertaining  to 
the:  (a)  collection  and  identification  of  data  prior  to  analysis;  (b)  specification  and  estimation  of  a 
regression  model;  and  (c)  interpretation  of  adjusted  cut  score  effects  on  attrition  rates. 

Data  Collection 

As  evident  from  the  preceding  discussion  (and  earlier  sections  of  this  report),  issues 
related  to  the  identification  and  collection  of  data  for  these  analyses  are  critical,  as  the  proposed 
methodology  will  only  be  as  good  as  the  available  data.  Our  recommendations  are  divided  into 
two  parts.  First,  we  make  recommendations  dealing  with  planning  an  attrition  rates  analysis 
(e.g.,  prior  to  actually  collecting  or  identifying  requisite  data).  Second,  we  make 
recommendations  related  to  the  collection  and/or  identification  of  data  for  analysis. 

Planning  an  Attrition  Rates  Analysis.  Prior  to  collecting  or  identifying  data  for  analysis,  it 
is  important  to  carefully  consider  school  sample  size  (. N)  and  magnitude  of  desired  change 
(increase  or  decrease)  in  attrition.  It  is  strongly  recommended  that  (any)  analyses  not  take  place 


26 


without  addressing  these  issues.  Considering  school  sample  size  and  magnitude  of  the  desired 
change  in  attrition  will  minimize  error  and  maximize  the  relative  precision  of  estimated  attrition 
rates.  Operationally,  this  will  enhance  the  decisions  made  by  Army  personnel  managers  and 
strategic  planners. 

With  respect  to  school  sample  size  (TV),  our  simulations  show  that  small  sample  sizes 
(vV<200)  are  problematic  because  estimated  attrition  rates  based  on  these  sample  sizes  are 
associated  with  a  large  amount  of  error.  As  a  result,  we  recommend  that,  where  possible, 
analyses  not  be  conducted  on  N’s  less  than  400.  For  best  results,  we  recommend  an  A  of  800. 
There  are  two  reasons  why  we  prefer  an  N  of  800.  First,  an  while  N  of  400  reduces  error  by 
about  50%  (from  an  N=  100),  an  N  of  800  cuts  error,  on  average,  by  two-thirds  (see  Figures  19 
to  27).  That  represents  a  considerable  gain  in  precision,  over  and  above  that  expected  for  an  N  of 
400.  Second,  and  related  to  this  first  point,  there  is  a  tradeoff  between  N  and  practical  data 
collection  and  quality  control  (QC)  issues.  Although,  the  bigger  the  N,  the  smaller  the  error, 
larger  sample  sizes  are  not  always  practically  feasible,  due  to  limited  time  and  resources.  In 
addition,  larger  sample  sizes  are  associated  with  increased  data  management  and  QC  issues  (e.g., 
more  time  spent  managing  data,  greater  percentages  of  missing  or  inaccurate  data).  For  these 
reasons,  there  is  a  point  of  diminishing  returns,  where  a  higher  N  does  not  yield  substantially  new 
information.  For  example,  for  sample  sizes  from  100  to  800,  every  increase  in  Nby  100 
produces  an  (average)  reduction  in  error  of  about  8%.  From  sample  size  800  to  1600,  the 
(average)  reduction  drops  to  1-2%.  For  sample  sizes  from  1600  to  3200,  the  (average)  reduction 
drops  even  further  to  less  than  1%  (for  every  increase  in  N  of  100).  This  trend  is  visible  in 
Figures  19  to  27.  As  evident  from  these  figures,  doubling  the  sample  size  results  in 
progressively  smaller  reductions  in  error,  going  from  roughly  30%  (N=  100  to  200)  to  20%  (N  = 
200  to  400)  to  15%  (A  =400  to  800)  to  10%  (N  =  800  to  1600)  to  7%  (N=  1600  to  3200).  In 
summary,  there  is  a  point  where  the  costs  of  greater  N  (e.g.,  time  collecting  data)  outweigh  the 
benefits  (e.g.,  smaller  error),  and  that  point  appears  to  be  around  an  N  of  800. 

Having  made  the  above  recommendations;  we  acknowledge  that  an  N  of  800  (or  even 
400)  may  not  always  be  feasible,  particularly  for  MOS  that  train  small  numbers  of  recruits  (N  < 
50)  per  year.  For  those  MOS,  we  suggest  the  following.  First,  we  advise  combining  data  from 
multiple  fiscal  years  (FYs)  to  achieve  a  preferred  level  of  N.  Second,  if  combining  multiple  FYs 
is  not  practical,  we  suggest  combining  data  from  comparable  MOS  (e.g.,  similar  training  and  job 
requirements).  The  procedures  for  identifying  comparable  MOS  are  beyond  the  scope  of  the 
present  study,  but  procedures  are  available  within  the  applied  personnel  psychology  literature  for 
synthesizing  data  from  multiple  jobs  for  purposes  of  making  personnel  decisions.  This  particular 
suggestion  holds  for  MOS  that  are.new  and  for  which  there  is  little  to  no  historical  data  from 
previous  FYs.  Third  and  finally,  should  less  than  preferred  levels  nf  iV be  available,  we 
recommend  that  school  proponents  and  Army  personnel  managers  be  conservative  when 
interpreting  estimated  attrition  rates. 

The  second  factor  to  consider  prior  to  performing  an  attrition  rates  analysis  is  the  desired 
change  (increase  or  decrease)  in  the  attrition  rate.  We  recommend  not  targeting  a  change  in 
attrition  arbitrarily.  This  is  important  because  the  magnitude  of  the  desired  change  (large  or 
small)  will  impact  how  high  or  low  the  cut  score  should  be  adjusted  to  achieve  the  desired 
attrition  rate.  The  size  of  the  prospective  training  pool  is  impacted  the  more  one  has  to  adjust  the 
cut  score  to  be  reasonably  confident  that  a  targeted  change  in  attrition  will  be  achieved.  From  an 
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operational  perspective,  this  is  not  trivial  since  inadvertently  excluding  enlistees  from  an  MOS’s 
training  pool  could  decrease  the  number  of  eligible  enlistees  in  the  pool  and  adversely  impact 
one’s  accession  goals.  For  instance,  using  the  earlier  example,  the  cut  score  needed  to  be 
adjusted  to  1 15  or  1 1 1  rf  using  samples  sizes  200  or  800,  respectively,  to  be  confident  that  the 
true  attrition  rate  is  not  more  than  10  percent.  Suppose  the  current  cut  score  is  104,  for  which 
approximately  42%  of  the  youth  population  is  eligible,11  Then  the  percentage  of  the  eligible  pool 
will  drop  by  19  percentage  points  (from  42%  to  23%)  if  the  cut  score  is  adjusted  to  1 15,  but  will 
only  drop  by  13  percentage  points  (from  42%  to  29%)  if  it  is  adjusted  to  1 1 1.  The  latter  is  closer 
to  the  minimum  possible  drop  of  six  percentage  points  (from  42%  to  36%)  if  we  have  perfect 
information  and  are  able  to  adjust  the  cut  score  to  107,  which  would  yield  the  10%  targeted 
attrition  rate. 

In  summary,  given  a  fixed  target  attrition  rate,  the  sample  size  (N)  chosen  for  analysis 
represents  a  trade-off  between  the  short-term  costs  associated  with  data  collection  and 
management  versus  the  long-term  savings  related  to  recruiting,  that  is  generally  made  possible 
by  not  having  to  substantially  increase  the  cut  score. 

Collecting  or  Identifying  Requisite  Data.  Once  the  school  sample  size  and  desired  change 
in  attrition  have  been  addressed,  data  can  be  collected,  or  alternatively,  existing  data  may  be 
identified  for  performing  the  attrition  rate  analysis.  The  issues  to  be  considered  at  this  stage  are: 
(a)  the  number  of  training  classes  to  include  in  the  analysis;  (b)  accuracy  of  data;  (c)  missing 
data;  and  (d)  criterion  contamination. 

First,  regarding  the  number  of  classes  to  include  in  the  analysis,  most  likely  data  will 
need  to  be  collected  from  multiple  classes  to  reach  recommended  levels  of  N  (e.g.,  400  -  800). 
Practically,  classes  currently  in  session  should  be  excluded  from  the  analysis.  An  important 
statistical  issue  related  to  multiple  classes  is  the  non-independence  of  observations  from  trainees 
belonging  to  the  same  class,  and  potentially,  across  classes.  This  non-independence  could  be  due 
to  a  variety  of  factors,  such  as  multiple  classes  having  the  same  instructor.  Non-independence,  if 
not  addressed,  could  underestimate  error.  The  present  methodology  does  not  take  this  into 
account,  but  future  research  on  this  issue  is  highly  recommended. 

Second,  with  respect  to  the  accuracy  of  collected  or  existing  data,  there  are  several  steps 
that  can  be  taken  to  ensure  data  integrity.  These  steps  are  extremely  important,  as  the  quality  of 
the  data  will  significantly  impact  the  results  of  any  analyses,  and  operationally,  decisions  made 
based  on  the  results.  Prior  to  data  collection,  it  is  recommended  that  efforts  be  made  to 
standardize  data  collection  and  minimize  potential  data  entry  errors.  These  could  include 
standardizing  data  entry  protocols,  The-development  of  a  single  database  to  which  all  data  will  be 
entered,  or  the  creation  of  fields  for  screening  data  for  errors  upon  entry  (where  the  distance 
between  the  original  data  source  and  the  recorded  data  is  shortest).  Post-data  collection,  it  is 
highly  recommended  to  perform  a  set  of  data  screening  (or  QC)  procedures.  These  procedures 


11  This  approximation  is  based  on  a  normally  distributed  AA  score  with  mean  equal  to  100  and  standard  deviation  of 
20. 
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would  include  checking  for  out-of-range  values  and  missing  data.12  If  data  screening  identifies 
significant  problems,  additional  data  collection  may  be  warranted.  In  general,  it  is  best  if  these 
problems  can  be  avoided  up  front  by  taking  steps  (see  above)  to  minimize  potential  problems  as 
data  are  collected.  In  the  long  run,  doing  so  will  substantially  reduce  the  time  and  resources 
expended  in  collecting  data. 

Third,  specific  to  missing  data,  those  performing  attrition  rates  analysis  are  advised  to 
check  the  percentage  of  cases  missing  data  and  to  examine  the  nature  of  missing  data  (e.g., 
random  versus  not  random).  Sizeable  percentages  of  missing  data  are  problematic  because  they 
could  contribute  to  estimation  problems  when  running  the  analysis.  Likewise,  data  not  missing 
at  random  could  seriously  bias  results,  as  the  cases  with  complete  data  may  differ  significantly 
from  those  missing  data.  Where  possible,  and  the  accuracy  of  doing  so  can  be  verified  using 
available  data,  it  is  recommended  that  missing  data  be  filled  in  prior  to  analysis.  This  may  not 
always  be  possible,  but  can  be  done  when  necessary  data  are  available.  For  example,  if  data  on 
output  codes  are  missing  for  some  cases  but  complete  training  performance  data  is  available,  the 
correct  output  code  can  be  imputed  based  on  the  existing  data  and  knowledge  of  the  cut  scores 
for  the  different  training  tests  (e.g.,  a  failing  grade  on  at  least  one  of  the  tests  results  in  the  trainee 
being  classified  as  “attrit”).  Generally,  it  is  preferable  if  missing  data  can  be  avoided  by  taking 
steps  to  minimize  incomplete  records  when  data  are  collected. 

Fourth,  those  performing  an  attrition  rates  analysis  are  likely  to  encounter  possible 
criterion  contamination.  This  occurs  when  a  trainee’s  output  code  is  inconsistent  with  training 
performance  data  (e.g.,  trainee  failed  one  or  more  tests,  but  output  code  indicates  that  the  student 
graduated).  There  are  several  reasons  why  criterion  contamination  may  be  present  (e.g.,  data 
entry  error,  trainee  warrants  special  exemptions).  Irrespective  of  the  reason,  criterion 
contamination  is  problematic  in  that  it  biases  results.  To  reduce  the  potential  bias  resulting  from 
criterion  contamination,  it  is  recommended  that:  (a)  non-academic  attritions  be  excluded  from 
the  analysis;  and  (b)  the  alignment  between  output  codes  and  training  performance  data  be 
checked  for  discrepancies  (e.g.,  output  code  indicates  trainee  graduated,  but  training  performance 
data  shows  that  he/she  did  not  pass  one  or  more  tests).  When  verifying  the  integrity  of  the  output 
codes,  check  if  the  percentage  of  discrepancies  matches  what  is  to  be  expected,  as  most  MOS 
will  probably  contain  some  percentage  of  trainees  passed  for  special  exemptions.  If  the 
percentage  of  cases  with  discrepancies  is  within  the  expected  percentage  (of  total  cases),  these 
problem  cases  can  simply  be  aligned  to  make  the  output  code  reflect  the  “true”  academic  training 
outcome.  If  the  percentage  of  cases  is  greater  than  expected,  the  integrity  of  this  information  is 
suspect  and  additional  data  collection  may  be  warranted. 

Interpreting  the  Effects  of  Adjusted  Cut  Scores  on  Attrition  Rates 

There  are  several  issues  that  should  be  considered  when  interpreting  the  effects  of 
adjusted  cut  scores  on  attrition  rates. 


~  See  Tabachnick  and  Fidell  (1996)  for  a  practical  and  comprehensive  overview  of  data  screening  procedures. 
See  Roth  (1998)  or  Switzerand  Roth  (2002)  for  guidelines  and  suggestions  on  dealing  with  missing  data. 
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First,  as  evident  from  the  results  of  our  simulation,  interpretation  must  account  for  the 
impact  of  school  sample  size  ( N)  on  the  accuracy  and  precision  of  observed  attrition  rates.  It  is 
strongly  recommended  that  this  factor  be  considered  when  interpreting  the  results  of  an  attrition 
rates  analysis,  as  it  could  substantially  influence  the  operational  decisions  made  by  Army 
personnel  managers  and  strategic  planners.  Accuracy  and  precision  considerations  also  apply 
when  contemplating  adjusted  cut  score  effects  on  personnel  outcomes  other  than  attrition  rates 
(e.g.,  demographic  representation).  For  example,  when  examining  other  personnel  outcomes  in 
conjunction  with  attrition  rates,  the  same  issues  (e.g,,  error,  magnitude  of  expected  change)  are 
pertinent  and  should  be  taken  into  account  when  interpreting  these  effects. 

Second,  it  should  be  noted  that  not  all  attrition  is  intrinsically  “bad.”  This  is  why  a  0% 
attrition  rate  is  not  a  meaningful  (or  practically  feasible)  operational  goal.  To  some  degree,  the 
seriousness  of  attrition  rates  depends  on  who  is  failing  to  complete  training.  For  example,  if 
those  failing  to  complete  training  are  poor  quality  recruits  not  likely  to  complete  their  first-term 
enlistment  or  satisfactorily  meet  minimum  performance  standards,  then  attrition  is  serving  a 
functional,  healthy  purpose.  It  is  possible  that  there  is  a  level  at  which  attrition  is  positively 
contributing  to  overall  mission  performance  (as  a  whole).  We  recommend  that  this  point  be 
considered  when  both  determining  the  expected  change  in  attrition  and  when  interpreting 
adjusted  cut  score  effects. 

Finally,  it  is  advisable  to  consider  if  changing  cut  score  is  always  the  best  available 
means  for  managing  academic  attrition.  Operationally,  altering  MOS  cut  score  is  just  one  of 
several  strategies  (e.g.,  adjusting  passing  grades,  revising  content  and  delivery  of  training, 
shortening  the  number  of  tests  required  to  graduate,  etc.)  for  minimizing  attrition.  As  seen  from 
our  simulation,  substantial  reductions  (e.g.,  33%-50%)  in  attrition  are  practically  not  feasible 
without  major  adjustments  in  minimum  enlistment  standards.  Therefore,  to  achieve  desired 
levels  of  attrition,  it  is  recommended  that  Army  personnel  managers  and  strategic  planners 
consider  combining  cut  score  adjustments  with  other  personnel  management  strategies.  This 
may  be  particularly  advantageous  when  lowering  the  cut  score  to  meet  accession  goals,  as 
decreases  in  enlistment  standards  are  associated  with  increases  in  attrition  rates.  It  may  be 
possible  to  mitigate  these  increases  by  implementing  proposed  cut  score  adjustments  in 
combination  with  other  strategies. 

Suggestions  for  Future  Research 

The  following  are  suggestions  for  future  research  that  would  further  enhance  our 
understanding  of  training  attrition  and  the  effects  of  changing  minimum  enlistment  standards  for 
the  Army  (as  a  whole).  These  suggestions  would  greatly  contribute  to  our  knowledge  of  factors 
contributing  to  attrition,  and  more  practically,  improve  the  decision-making  of  Army  personnel 
managers  and  strategic  planners. 

Expanding  the  Model  for  Informing  Related  Personnel  and  Training  Decisions 

For  the  present  study,  the  primary  goal  was  to  model  the  effects  of  changes  in  enlistment 
standards  on  academic  attrition  for  purposes  of  informing  the  setting  of  these  standards. 
However,  Army  school  proponents,  personnel  managers,  and  strategic  planners  may  be 
interested  in  extending  the  current  model  to  inform  other  related  personnel  and  training 


30 


decisions.  For  example,  as  evident  from  our  results,  for  some  MOS,  understanding  the  effects  of 
enlistment  standards  on  attrition  may  not  be  especially  informative  for  determining  how  best  to 
minimize  attrition.  Therefore,  school  proponents  of  those  MOS  may  be  interested  in 
understanding  the  effects  on  attrition  when  qualification  standards  other  than  cognitive  aptitude 
(e.g.,  medical  status)  are  adjusted.  Likewise,  strategic  planners  may  be  interested  in  assessing 
the  effects  of  changes  in  enlistment  standards  on  accession  goals  for  different  demographic 
groups.  Both  of  these  examples,  along  with  others,  could  be  addressed  by  extending  the  current 
model  to  include  variables  other  than  cognitive  aptitude.  The  following  are  variables  that  could 
be  added  to  the  current  model  for  purposes  of  supporting  personnel  and  training  decisions  related 
to  the  current  model’s  purpose. 

While  cognitive  aptitude  is  a  good  predictor  of  training  performance  (e.g.,  Earles  &  Ree, 
1992;  Hunter  &  Hunter,  1984;  Olea  &  Ree,  1994;  Ree  &  Earles,  1991),  in  many  cases  failure  to 
complete  training  is  not  a  function  of  lack  of  ability  (Ree  &  Carretta,  1999).  One  candidate  that 
might  account  for  attrition  beyond  cognitive  aptitude  is  educational  status.  Historically,  military 
studies  show  that  educational  status,  specifically  high  school  completion  status,  is  the  single  best 
predictor  of  first-term  attrition,  even  after  controlling  for  age,  cognitive  aptitude,  and  other 
personal  characteristics  (Laurence,  1984, 1987;  Laurence,  Naughton,  &  Harris,  1995).  There  are 
two  possible  explanations  for  this. 

First,  the  types  of  competencies  and  skills  necessary  to  complete  a  regular  high  school 
diploma  (or  two  full  years  of  college)  may  be  similar  to  the  skills  predictive  of  training  success. 
By  earning  a  regular  high  school  diploma,  or  other  educational  credential  requiring  a  similar 
degree  of  discipline  and  persistence  (e.g.,  college),  an  individual  has  demonstrated  the  ability  to 
learn  in  a  formalized  and  highly  structured  environment,  an  environment  comparable  to  that  of 
Army  training  programs. 

A  second  potential  explanation  for  the  relationship  between  educational  status  and 
attrition  is  that  educational  completion  status  is  a  surrogate  for  attributes  besides  cognitive 
aptitude,  such  as  motivation  or  personality.  An  extensive  body  of  research  shows  that  these 
characteristics  are  predictive  of  job  and  training  performance  across  a  variety  of  occupations 
(Barrick  &  Mount,  1991;  Barrick,  Mount,  &  Judge,  2001;  Hough  &  Fumham,  2003;  Judge  & 
Ilies,  2002).  In  addition,  there  is  strong  evidence  that  these  characteristics  meaningfully  add  to 
the  prediction  of  job  and  training  performance  over  and  above  cognitive  aptitude  (Hough  & 
Fumham,  2003;  McHenry,  Hough,  Toquam,  Hanson,  &  Ashworth,  1990;  Schmidt  &  Hunter, 
1998).  These  findings  suggest  that  assessing  characteristics  beyond  cognitive  aptitude  could 
greatly  contribute  to  predicting  training  performance. 

Medical  and  health-related  factors  constitute  another  potential  candidate  for  inclusion  in 
models  of  training  attrition  (Laurence  et  al.,  1995).  For  example,  in  a  recent  study  with  Navy 
recruits,  researchers  found  that  self-reported  medical  and  health-related  factors  were  stronger 
predictors  of  first-term  attrition  than  educational  status  or  cognitive  aptitude  (Larson,  Booth- 
Kewley,  &  Ryan,  2002).  This  finding  is  consistent  with  recent  U.S.  government  reports  on 
military  screening  practices  emphasizing  the  role  of  medical  and  health-related  factors  in  attrition 
(U.S.  General  Accounting  Office,  1997;  1998;  2000).  These  factors  may  be  especially  pertinent 
to  predicting  success  in  MOS  requiring  high  levels  of  physical  and  mental  endurance,  and  for 
predicting  attrition  during  training  due  to  non-academic  reasons. 
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In  summary,  expanding  the  current  model  to  include  factors  beyond  cognitive  aptitude  as 
predictors  could  greatly  benefit  Army  personnel  managers  and  strategic  planners.  Extending  the 
model  will  aid  decisions  on  how  best  to  reduce  attrition,  particularly  when  changes  in  enlistment 
standards  may  be  insufficient  for  achieving  a  targeted  level  of  attrition. 

Predicting  Trainee  Re-Test  Performance 

Operationally,  trainees  have  the  opportunity  to  retake  tests  that  they  failed  to  pass  on  the 
first  attempt.  Specifically,  trainees  are  permitted  a  second  attempt  for  all  required  tests  not 
passed  when  first  administered.  The  present  study  simulated  retest  behavior  assuming 
independent  test  attempt,  as  we  were  unable  to  examine  the  factors  contributing  to  retest 
performance.  Knowledge  of  factors  explaining  retest  performance  could  be  incorporated  into  the 
attrition  model  and,  consequently,  improve  its  predictive  performance. 

Possible  factors  that  account  for  retest  success,  include  those  factors  mentioned  above 
(e.g.,  education,  motivation,  personality,  medical/health-related  variables),  plus  variables  such  as 
self-efficacy.  Self-efficacy  reflects  an  individual’s  beliefs  about  his/her  task  capabilities,  and  has 
been  shown  to  be  a  powerful  predictor  of  learning  and  training  performance  (Cannon-Bowers, 
Salas,  Tannenbaum,  &  Mathieu,  1995;  Gist,  Schwoerer,  &  Rosen,  1989;  Martocchio,  1994; 
Mathieu,  Martineau,  &  Tannenbaum,  1993;  Salas  &  Cannon-Bowers,  2001).  Research  shows 
that  features  of  the  training  environment  (e.g.,  method,  content,  instructor),  in  particular  the 
feedback  trainees  receive,  can  contribute  to  a  trainee’s  self-efficacy  and  subsequent  performance 
(Cannon-Bowers  et  al.,  1995;  Eden  &  Ravid,  1982;  Gist  et  al.,  1989;  Mathieu  et  al.,  1993). 
Practically,  this  is  valuable  to  know,  since  in  the  long  run,  it  may  be  more  cost-efficient  for  the 
Army  to  develop  and  implement  guidelines  for  delivering  effective  training  feedback  Army-wide 
(and  thereby,  benefit  the  Army  as  a  whole),  as  opposed  to  instituting  changes  in  minimum 
enlistment  standards  for  individual  MOS.  Further,  as  self-efficacy  has  been  identified  as  a  means 
for  reducing  group  differences  in  test  performance  (e.g.,  Hough,  Oswald,  &  Ployhart,  2001; 
Sackett,  Schmitt,  Ellingson,  &  Kabiri,  2001),  these  strategies  may  prove  beneficial  in  achieving 
the  demographic  representation  Army  personnel  managers  seek  for  targeted  MOS,  without 
sacrificing  overall  performance. 

Modeling  Adjustments  to  Minimum  Enlistment  Standards  from  a  Multiple  Job  Context 

Most  training  research,  the  present  study  included,  produce  results  that  are  framed  in  a 
single  job  context.  How  well  these  findings  apply  to  a  multiple  job  context  (e.g.,  Zeidner  et  al., 
1997),  such  as  the  Army’s,  is  less  clear.  Future  research  would  benefit  from  investigating  and 
modeling  issues  of  interest  to  Army  personnel  managers  (e.g.,  setting  minimum  enlistment 
standards)  from  a  multiple  job  context.  In  particular,  future  research  that  examines  these  issues 
from  a  more  macro  perspective,  like  that  evident  in  classification  research  (e.g.,  Johnson  et  al., 
1992;  Zeidner  et  al.,  2000),  would  be  informative.  For  example,  future  research  should  consider 
how  adjustments  in  cut  scores  for  one  MOS  impact  the  attrition  rates  of  other  MOS.  Related  to 
this  work  would  be  the  development  and  validation  of  methodologies  for  maximizing  a  criterion, 
comparable  to  overall  performance  (e.g.,  MPP)  that  characterizes  military  classification  research. 
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Modeling  the  Relationship  between  Training  and  On-the-Job  Performance 

Future  research  might  consider  the  potential  tradeoffs  between  training  attrition  and 
subsequent  job  performance  for  the  Army  as  a  whole.  These  tradeoffs  (and  their  effects)  are 
presently  missing  in  both  Army  training  and  classification  research.  For  instance,  setting  high 
enlistment  standards  will  likely  increase  the  percentage  of  rejected  applicants  (for  the  Army  as  a 
whole),  but  will  also  likely  increase  overall  job  performance,  as  the  more  capable  Soldiers  are 
retained  and  receive  high  quality  training.  Alternatively,  lowering  enlistment  standards  may 
reduce  the  percentage  of  rejection,  but  at  the  same  time  negatively  impact  overall  performance, 
by  increasing  the  retention  of  poor  performers  and  reducing  the  overall  quality  of  training  that  all 
recruits  receive. 

Methodologies  that  model  these  potential  tradeoffs  could  greatly  assist  decision-makers 
in  identifying  strategies  that  satisfy  operational  goals  associated  with  both  (e.g.,  training  and 
solder  performance  on  the  job).  For  example,  by  lengthening  or  altering  the  content  of  its 
training  programs,  it  might  be  possible  for  the  Army  to  both  control  attrition,  while  maximizing 
overall  job  performance.  Practically,  this  could  also  translate  into  lower  enlistment  standards 
making  it  possible  for  the  Army  to  increase  the  eligible  accession  pool  and  meet  accession  goals 
without  the  associated  trade-offs  (e.g.,  Soldier  performance).  From  an  operational  perspective, 
these  methodologies  would  be  especially  important  when  changes  in  the  Army’s  recruiting 
environment  make  it  seem  that  potential  tradeoffs  between  training  effectiveness  and  Soldier 
performance  cannot  be  avoided. 


Conclusion 

Army  personnel  managers  frequently  need  to  make  tradeoffs  between  Soldier  numbers, 
quality,  training  effectiveness,  and  a  host  of  other  factors  when  making  personnel  management 
and  training  decisions.  The  purpose  of  the  present  study  was  to  propose  and  demonstrate  a 
logistic  regression-based  approach  for  estimating  academic  attrition  rates  from  technical  training 
school.  This  approach  enables  Army  personnel  managers  and  strategic  planners  to  evaluate  the 
aforementioned  tradeoffs  when  making  decisions  about  where  to  set  minimum  enlistment 
standards.  To  demonstrate  the  approach,  we  conducted  a  large-scale  simulation  study  that 
simulated  attrition  rates  under  different  operational  scenarios  Army  managers  are  likely  to 
encounter  on  the  job.  SAS  programs  (with  documentation)  for  running  an  attrition  rates  analysis 
on  MOS  school  data  using  the  proposed  approach  are  available  in  Appendix  E. 

The  major  findings  of  our  simulation  are  threefold.  First,  a  simple  approach  based  on 
logistic  regression  using  only  cognitive  aptitude  information  is  adequate  for  evaluating  the 
impact  of  changes  in  minimum  enlistment  standards  on  academic  attrition  for  MOS  with  medium 
complexity/validity  or  greater.  For  some  MOS,  especially  those  with  low  difficulty  or  low 
complexity/validity,  reducing  or  reaching  a  desired  level  of  academic  attrition  may  be  best 
achieved  by  methods  other  than  adjustments  in  minimum  enlistment  standards. 

Second,  school  sample  size  (N),  on  which  academic  attrition  estimates  are  based,  can 
significantly  impact  the  quality  of  the  decisions  made  based  on  these  estimates.  Operationally, 
this  has  implications  about  whether do  and  how  much  to  adjust  enlistment  standards  to  achieve 
desired  objectives.  A  cost-benefit  analysis  indicated  that  a  sufficiently  large  sample  size  allows 
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smaller  changes  in  minimum  enlistment  standards  in  order  to  achieve  a  targeted  attrition  rate 
with  high  confidence,  which  in  turn  translates  to  potential  savings  in  terms  of  the  size  of  the 
eligible  applicant  pool.  Specifically,  we  observed  that  larger  sample  size  (N  >  400)  is  more 
likely  to  produce  more  accurate  estimates  and  thereby,  better  quality  decisions. 

Third,  personnel  management  and  training  decisions  could  be  greatly  improved  in  some 
cases  by  incorporating  information  other  than  cognitive  aptitude.  For  example,  as  evident  from 
our  results,  for  some  MOS,  understanding  the  effects  of  enlistment  standards  on  attrition  may  not 
be  especially  informative  for  determining  what  strategy  is  best  for  minimizing  attrition. 
Therefore,  school  proponents  of  those  MOS  may  be  interested  in  understanding  the  effects  on 
attrition  from  adjusting  enlistment  standards  (e.g.,  educational  requirements)  other  than  cognitive 
aptitude.  In  particular,  MOS  with  low  difficulty  or  low  complexity/validity  could  benefit  from 
extending  the  current  model  with  this  additional  information. 


There  are  a  number  of  practical  recommendations  for  performing  and  interpreting  this 
analysis  that  would  aid  operational  decisions.  We  presented  these  recommendations  and 
suggested  several  areas  for  future  research  that  could  greatly  extend  the  proposed  approach. 
Research  in  these  suggested  areas  would  further  enhance  the  effectiveness  of  the  difficult,  but 
strategically  important,  decisions  Army  managers  must  make. 
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APPENDIX  A:  NOTES  ON  SIMULATING  AA  COMPOSITE  AND  TRAINING 

PERFORMANCE  DATA 
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Notes  on  Simulating  AA  Composite  and  Training  Performance  Data 
Converting  OldAA  to  New  AA  Scores 

Note  that  conversion  of  old  AA  scores  is  only  an  issue  if  the  original  scores  on  the  nine 
ASVAB  subtests  are  no  longer  available.  If  original  scores  for  the  ASVAB  subtests  are 
otherwise  unavailable,  conversion  from  old  to  new  AA  scores  can  be  carried  out  using  the  matrix 
expression 


—new  ^  new  ifLold  +  E-olcl  )J—  K.mw 

In  the  expression  above,  X 0u  is  the  vector  of  nine  old  AA  scores;  Xev.  is  the  vector  of  nine  new 
AA  scores;  Uo!d  is  the  9x9  matrix  whose  rows  are  the  vectors  of  conversion  weights  and  Kotd  is  a 
9x1  vector  of  constants  for  the  old  unit  weighted  AA  score;  Unew  is  the  9x7  matrix  whose  rows 
are  the  vectors  of  conversion  weights  and  K„ew  is  a  9x1  vector  of  constants  for  the  new  AA 
composites;  finally  D  is  the  7x9  matrix  constructed  by  deleting  the  rows  in  the  9x9  identity 
matrix  corresponding  to  Numerical  Operation  (NO)  and  Coding  Speed  (CS)  subtests.13  Note  that 
the  expression  inside  the  parentheses  is  just  the  vector  of  seven  ASVAB  subtests  (with  NO  and 
CS  dropped). 

Conceptually,  this  formula  is  essentially  de-standardizing  the  old  unit-weighted  AA 
(standard)  scores  to  the  original  sum  of  ASVAB  subtest  scores,  then  applying  the  new 
conversion  weights  and  constants  to  those  scores  to  obtain  the  new  AA  (standard)  scores 
currently  in  operational  use  by  the  Army  (Zeidner  et  al.,  2000).  To  verify  the  accuracy  of  this 
procedure  and  of  the  conversion  weights  for  moving  from  the  old  unit-weighted  AA  standard 
cores  to  the  sum  of  ASVAB  subtest  scores,  we  did  the  following. 

First,  we  obtained  historical  information  for  converting  the  sum  of  ASVAB  subtest  scores 
to  the  old  unit-weighted  AA  standard  scores  (U.S.  Department  of  Defense,  1990).  Second,  we 
regressed  unit-weighted  AA  scores,  excluding  the  lowest  and  highest  possible  values  of  40  and 
160,  onto  their  respective  sum  of  ASVAB  subtest  scores  (based  on  the  conversion  information) 
for  all  9  AA  composites.14  The  resulting  regression  parameters  (intercepts,  regression  weights) 
represented  the  conversion  weights  and  constants  (e.g.,  Uo!d  and  Kold)  for  converting  from  the  old 
unit-weighted  AA  standard  scores  to  the  sum  of  ASVAB  subtest  scores  for  all  9  AA  composites. 
These  values  are  reported  in  Table  1  (at  end  of  Appendix  A).  For  convenience,  Table  2  contains 
the  conversion  weights  (Umw)  and  constants  (KneK)  for  generating  the  new  AA  standard  scores. 


13  For  example,  if  NO  and  CS  correspond  to  the  third  and  fourth  subtests  in  X iM  then  delete  the  third  and  fourth 
rows  in  the  9x9  identity  matrix  to  obtain  the  7x9  matrix  D. 

W  e  excluded  40  and  1 60  as  these  values  reflect  the  minimum  and  maximum  conversion  values  for  converting 
ASVAB  subtest  scores  to  unit-weighted  AA  scores,  and  effectively  represent  all  AA  scores  below  or  above  40  and 
160  respectively. 
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Third,  upon  having  generated  the  conversion  weights,  we  validated  the  weights  by 
comparing  the  predicted  AA  standard  score  (using  the  previously  generated  regression 
parameters)  to  the  corresponding  AA  standard  score  from  which  the  predicted  scores  were  based. 
Across  all  9  composites,  only  7  cases  exhibited  residuals  greater  than  0.5.  Of  these  7  cases,  the 
difference  between  the  tabulated  AA  scores  and  AA  scores  estimated  using  our  procedure, 
rounded  to  the  nearest  integer,  was  no  greater  than  1 .0.  In  summary,  for  the  purposes  of 
estimating  attrition  rates,  there  is  strong  support  for  the  accuracy  of  our  procedure  and  of  the 
obtained  conversion  weights  and  constants  (see  Table  1)  for  converting  old  unit- weighted  AA 
standard  scores  to  new  AA  standard  scores. 

Procedure  for  Generating  Synthetic  AA  and  Training  Performance  Data 

When  simulating  AA  composite  and  training  performance  data,  our  ultimate  goal  was  to 
generate  synthetic  data  consistent  with  training  requirements  and  testing  process  of  the  typical 
MOS.  In  brief,  the  testing  process  in  the  typical  MOS  school  can  be  characterized  as  a  multiple 
hurdle  system.  Students  are  required  to  pass  each  test  in  the  sequence  before.they  can  move  on 
to  the  next  test.  Passing  a  test  means  obtaining  a  grade  that  is  equal  to  or  better  than  the 
minimum  passing  grade  for  the  test,  which  could  (and  does)  vary  from  one  test  to  another. 
Generally,  a  student  who  fails  a  given  test  is  allowed  one  opportunity  to  retrain  and  retake  the 
test.  Therefore,  according  to  Army  policy,  students  have  at  most  two  attempts  to  pass  each  test 
in  the  sequence.  If  they  fail  a  test  in  a  sequence  (i.e.,  scoring  below  the  passing  grade  on  the 
second  attempt),  then  the  student  is  dropped  from  the  class.15 

To  model  these  features  we  did  the  following.  Let  T  be  equal  to  the  total  number  of  tests 
that  trainees  must  pass  to  successfully  complete  the  training.  Using  i  and  j  to  respectively  index 
the  trainee  and  test,  the  vector  of  test  scores  for  the  z'th  trainee  is  represented  by  (Fn,  Fi2, FiT}. 
The  relationship  between  the  test  score  Fy  on  one  attempt  and  the  AA  score  X\  is  assumed  to  be 
given  by  regression 


Yu  ~  aj  +  bjXi  +  £ij 

Conditional  on  the  AA  score  Aj,  the  regression  errors  etJ  are  assumed  to  be  independent  across 
trainee  given  the  jth  test,  but  are  correlated  across  tests  for  a  particular  trainee.  That  is, 
cor[£0  ,erf)  equals  zero  if  i  *  /'  for  any  j  and  j’,  but  is  possibly  non-zero  for  i  =  f . 

The  above  specification  completely  describes  the  mean  and  covariance  structure  of  the 
test  scores  indicative  of  trainee  performance.  The  overall  structure  of  the  multivariate  outcome 
variable  of  test  scores  conditional  on  the  AA  score  can  be  described  in  terms  of  the  regression 
parameters,  the  intercept  and  coefficient  (aj,bj),  validities  rf  =  cor(fjj,Xj )  and  intra-person  error 

correlation  cor^^s^., ).  The  different  operational  scenarios  used  in  our  simulation  experiments 
represent  different  combinations  of  these  three  sets  of  parameters.  In  our  simulation,  we 


l5  When  dropped  from  a  class,  the  student  may  be  recycled  into  the  same  MOS  school,  an  alternative  MOS  school, 
Dr  discharged  from  the  Army. 
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included  trainee  re-test  behavior  (from  being  allowed  two  attempts  to  pass  a  test),  by  assuming 
that  the  two  attempts  for  given  applicant  are  independent.16 


i6  Given  that  retraining  of  applicants  who  failed  the  first  attempt  likely  is  designed  to  give  them  a  better  chance  of 
passing  the  second  attempt,  the  assumption  of  independence  between  two  attempts  would  tend  to  lead  to  slightly 
more  conservative  attrition  estimates.  Note  that  estimation  of  the  between-attempts  correlation  would  require  test 
scores  from  both  attempts.  Adequate  sample  size  may  be  difficult  to  achieve  for  this  purpose. 
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Table  1 

Conversion  Weights  (U0id)  and  Constants  (Koid)  for  Converting  Unit-Weighted  AA  Standard  Scores  to  ASVAB  Subtest  Scores 
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Table  1.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  Low  Difficulty,  Low  Complexity  Condition 


Cut 

Scores 

Alternative  Sample  Sixes 

N  = 

100 

N  = 

200 

N  * 

400 

N* 

800 

N* 

1600 

N* 

3200 

AVG 

STD 

AVG 

ST1) 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

GM=  80 

0.48 

0.078 

0.49 

0.057 

0.49 

0,041 

0.49 

0.029 

0.49 

0,020 

0,49 

0.015 

GM=  81 

0.48 

0.077 

0.48 

0.057 

0.48 

0,040 

0,48 

0.029 

0,48 

0.020 

0.48 

0.014 

GM*  82 

0.47 

0.076 

0.48 

0.056 

0.48 

0.040 

0.48 

0.028 

0.48 

0.020 

0.48 

0.014 

GM=  83 

0.47 

0.075 

0.47 

0.055 

0.48 

0.039 

0.47 

0.028 

0.47 

0.019 

0.47 

0.014 

GM*  84 

0.47 

0.074 

0.47 

0,054 

0.47 

0.039 

0,47 

0.028 

0.47 

0.019 

0,47 

0.014 

GM-  85 

0.46 

0.073 

0.46 

0.053 

0.47 

0,038 

0.46 

0,027 

0.46 

0.019 

0,46 

0.013 

GM=  86 

0.46 

0.071 

0.46 

0,052 

0.46 

0.037 

0,46 

0.027 

0,46 

0,018 

0,46 

0.013 

GM®  87 

0.45 

0.45 

0.037 

0.45 

0.026 

0.45 

0.018 

0.45 

0.013 

GM-  88 

0.45 

0.068 

0.45 

0.050 

0.45 

0,036 

0.45 

0.025 

0.45 

0.018 

0.45 

0.013 

GM=  89 

0.44 

0.067 

0.44 

0.049 

0.44 

0.035 

0.44 

0,025 

0,44 

0.017 

0.44 

0,012 

GM=  90 

0.43 

0.065 

0.44 

0,048 

0,44 

0.034 

0.43 

0.024 

0,43 

0.017 

0.43 

0,012 

GM=  91 

0.43 

0.064 

0.43 

0.046 

0.43 

0.033 

0.43 

0.024 

0.43 

0.016 

0.43 

0.012 

GM=  92 

0.42 

0.062 

0.42 

0,045 

0.42 

0,033 

0.42 

0.023 

0.42 

0.016 

0.42 

0.011 

GM=  93 

0.41 

0.061 

0.42 

0.044 

0.42 

0.032 

0,41 

0.022 

0.42 

0.016 

0,42 

0.011 

GM=  94 

0.41 

0.059 

0.41 

0.042 

0.41 

0,031 

0.41 

0,022 

0.41 

0,015 

0.41 

0.011 

GM=  95 

0.40 

0.057 

0.40 

0.041 

0.40 

0,030 

0.40 

0.021 

0.40 

0.015 

0.40 

0.010 

GM=  96 

0.39 

0.056 

0.39 

0.040 

0.40 

0,029 

0.39 

0.020 

0.39 

0.014 

0.39 

0.010 

GM=  97 

0.39 

0.054 

0.39 

0.039 

0.39 

0.028 

0,39 

0.020 

0.39 

1  0.014 

j 

0.39 

0.010 

GM=98 

0.38 

0.052 

0,38 

0.037 

0.38 

0.027 

0.38 

0.019 

0,38 

0.013 

0.38 

0.009 

GM=  99 

0.37 

0.051 

0.37 

0.036 

0.37 

0,027 

0.37 

0.018 

0.37 

0,013 

0,37 

0.009 

GM*100 

0.36 

0.050 

0,36 

0.035 

0.37 

0.026 

0.36 

0.018 

0.36 

1  0,012 

0.36 

0.009 

GM-101 

0.36 

0.049 

0.36 

0.034 

0.36 

0.025 

0.36 

0.017 

0,36 

0.012 

0,36 

0.009 

GM-102 

0.35 

0.048 

0.35 

0.033 

0.35 

0.025 

0.35 

0.017 

0.35 

0.012 

0.35 

0.008 

GM=103 

0.34 

0.047 

0.34 

0.033 

0,34 

0.024 

0.34 

0.016 

0.34 

0.012 

0.34 

0,008 

GM=104 

0.33 

0.046 

0.33 

0.032 

0.33 

0.024 

0.33 

0.016 

0.33 

0.011 

0.33 

0.008 

GM=105 

0.33 

0.046 

0.33 

0,032 

0,33 

0.024 

0.32 

0,016 

0.32 

0.011 

0.32 

0.008 

GM-106 

0.32 

0.046 

0.32 

0.032 

0.32 

0.023 

0.32 

0.016 

0.32 

0.011 

0.32 

0.008 

GM-107 

0.31 

0.046 

0.31 

0.032 

0,31 

0.023 

0.31 

0.016 

0.31 

0.011 

0.3  i 

0.008 

GM=108 

0.30 

0.046 

0.30 

0.032 

0.30 

0.024 

0.30 

0.016 

0,30 

0.011 

0.30 

0.008 

GM=109 

0.29 

0.047 

i 

0.29 

0.032 

0.29 

0.024 

0.29 

0.016 

0,29 

0.011 

0.29 

0.008 

Table  1.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  Low  Difficulty,  Low  Complexity  Condition 


”  Cul 
Scores 

Alternative  Sample  Sizes 

N  -  100 

NA;2<H) 

N  =  400 

N  =  800 

N  -  1600 

N  =  3200 

■■ 

AVG 

STD 

AVG 

STD 

AVG 

SID 

AVG 

STD 

AVG 

STD 

AVG 

STD 

GM=I  10 

0.29 

0.048 

0.28 

0.033 

0.28 

0.024 

0.28 

0.016 

0.28 

0.012 

0.28 

0.008 

GM--J  M 

0.28 

0.049 

0.28 

0.034 

0.28 

0.025 

0.28 

0.016 

0.28 

0.012 

0.28 

0.008 

GM-I12 

0.27 

0.050 

0.27 

0.034 

0.27 

0.025 

0.27 

0.017 

0.27 

0.012 

0.27 

0.008 

GM=fl3 

0.26 

0.051 

0.26 

0.035 

0.26 

0.026 

0.26 

0.017 

0.26 

0.012 

0.26 

0.009 

GM=#fl4 

0.25 

0.053 

0.25 

0.036 

0.25 

0.026 

0.25 

0.018 

0.25 

0.013 

0.25 

0.009 

GM«115 

0.25 

0.054 

0.25 

0.037 

0.25 

0.027 

0.25 

0.018 

0.25 

0.013 

0.24 

0.009 

GM=JJ6 

0.24 

0.056 

0.24 

0.038 

0.24 

0.027 

0.24 

0.019 

0.24 

0.013 

0.24 

0.009 

GM=ll7 

0.23 

0.057 

0.23 

0.040 

0.23 

0.028 

0.23 

.  . 

0.019 

0.23 

0.014 

0.23 

0.009 

GM-1 18 

0.23 

0.058 

0.22 

0.041 

GM=U9 

0.22 

0.060 

0.22 

0.042 

0.22 

0.029 

0.22 

0.020 

0.22 

0.014 

0.21 

0.010 

GM-120 

0.21 

0.061 

0.21 

0.043 

0.21 

0.030 

0.21 

0.021 

0.21 

0.015 

0.21 

0.010 

Table  2.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  Low  Difficulty,  Medium  Complexity  Condition 


Table  2.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  Low  Difficulty,  Medium  Complexity  Condition 


Alternative  Sample  Sizes 


cut 

Scores 

JV  = 

100 

N- 

200 

N  - 

400 

N  = 

800 

•N« 

1600 

N  =  3200 

\  A.  'f 

AVG 

STD 

AVG.. 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

GM=1 10 

0.04 

0.020 

0.04 

0.014 

0.04 

0.009 

0.04 

0.007 

0.04 

0.005 

0.04 

0.003 

GM=11I 

0.03 

0.020 

0.03 

0.014 

0.03 

0.009 

0.03 

0.006 

0.03 

0.005 

0.03 

0.003 

GM=112 

0.03 

0.020 

0.03 

0.014 

0.03 

0.009 

0.03 

0.006 

0.03 

0.005 

0.03 

0.003 

GM*113 

0.03 

0.020 

0.03 

0.013 

0.03 

0.009 

0.03 

0.006 

0.03 

0.005 

0.03 

0.003 

GM-114 

0.03 

0.020 

0.03 

0.013 

0.03 

0.009 

0.03 

0.006 

0.03 

0.005 

0.03 

0.003 

GM=115 

i  0.03 

0.021 

0.03 

0.013 

0.03 

0.009 

0.03 

0.006 

0.03 

|  0.005 

0.03 

0.003 

GM-116 

0.03 

0.021 

0.03 

0.013 

0.02 

0.009 

0.02 

0.006 

0.02 

0.004 

0.02 

0.003 

GM-117 

0.03 

0.021 

0.02 

0.013 

0.02 

0.009 

0.02 

0.006 

0.02 

0.004 

0.02 

!  0.003 

GM-118 

0.02 

0.022 

0.02 

0.013 

0.02 

0.009 

0.02 

0.006 

!  0.02 

i  0.004 

0.02 

0.003 

jGM^I  19 

0.02 

0.022 

0.02 

0.013 

0.02 

0.009 

0.02 

0.006 

0.02 

0.004 

0.02 

o 

b 

o 

u> 

GM-120 

0.02 

0.022 

0.02 

0.013 

0.02 

0.008 

0.02 

0.006 

0.02 

0.004 

0.02 

0.003 
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Table  3.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  Low  Difficulty,  High  Complexity  Condition 


Cut 

Scores 


Alternative  Sample  Sizes 


N  -  100 
A  VG I  STD 


GM=  SO 
GM=  81 
GM=  82 
GM-  83 
GM=  84 
GM-  85 
GM-  86 
GM=  87 
GM-  88 
GM-  89 
GM=  90 
GM=9I 
GM=  92 
GM-  93 
GM=  94 
GM=  95 
GM-  96 
GM-  97 
GM=  98 
GM=  99 
GM-100 
GM-I01 
GM-IQ2 
GM-103 
GM=104 
GM-I05 
GM-106 
GM— 107 
GM-108 
GM=I09 


0.23  0.117 
0.22  0.114 
0.22  0.112 
0.21  0.109 
0.20  0.106 
0.19  0.102 
0.18  0.099 
0.16  0.094 
0.15  0.090 
0.14  0.085 
0.13  0.080 
0.12  0.074 
0.11  0.069 
0.09  0.063 
0.08  0.056 
0.07  0.050 
0.06  0.043 
0.05  0.037 
0.04  0.031 
0.03  0.026 
0.03  0.021 
0.02  0.018 
0.02  0.014 
0.02  0.012 
0.01  0.010 
0.01  0.009 
0.01  0.008 
0.01  0.007 
0.01  0.007 
0.01  0.006 


.  N  = 

200 

AVG 

SID 

0.20 

0,096 

0,19 

0.094 

0,19 

0.091 

0.18 

0.088 

0.17 

0.085 

0.16 

0.082 

0.15 

0.078 

0.14 

0.074 

0,12 

0.069 

0,11 

0.064 

0.10 

0,059 

0.09 

1  0.054 

0.08 

0.049 

0.07 

0,043 

0.06 

0.038 

0.05 

0.033 

0.05 

0,028 

0.04 

0,024 

0.03 

0,020 

0,03 

0.016 

0.02 

0.014 

0.02 

0,012 

0.02 

0,010 

0.01 

0.008 

0.01 

0.007 

0.01 

0.006 

0.01 

0.006 

0,01 

0.005 

0.01 

0.005 

0.01 

o 

o 

o 

4iv 

N  =  1 600 

avgIstp  . 

0.19  0.037 
0.18  0.036 

0.17  0.035 

I - 

0,36  0,034 
0.15  0,033 
0.14  0,031 
0.13  0.029 
0.12  0.027 
0.11  0.025 
0.10  0,023 
0,09  0.021 
0,08  0.019 
0.07  0,016 
0.06  0.014 
0.05  0.012 
0.05  0.010 
0,04  0.009 
0,04  0,008 
0.03  0.006 
0.03  0.005 
0.02  0.005 
0.02  0.004 
0.02  0.003 
0.01  0,003 
0.01  0.003 
0,01  0.002 
0.01  0,002 
0.01  0,002 
0.01  0.002 
0.00  0,002 


N  ~  3200 
AVG !  STD 


0,19  0.027 
0.18  0.027 
0.17  0.026 
0.16  0.025 
0.15  0.024 
0,14  0.023 
0.13  0.022 
0.12  0.020 
0.11  0.019 
0.10  0.017 
0.09  0.015 
0.08  0,014 
0.07  0.012 
0,06  0.010 
0.05  0,009 
0,05  0.008 
0.04  0.006 
0.04  0,005 
0.03  0.005 
0.03  0.004 
0.02  0.003 
0.02  0,003 
0.02  0,002 
0.01  0.002 
0.01  0,002 
0.01  0.002 
0.01  0.002 
0.01  0,001 
0.01  0.001 
0.00 1  0.001 


Table  3.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  Low  Difficulty,  High  Complexity  Condition 


Cut 

Scores 

Alternative  Sample  Sizes 

N  = 

10(1 

N  ~ 

200 

N  ™ 

:N  = 

1600 

N  =  3200 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

GM=110 

0.01 

GM=1 1 1 

0.00 

l^n 

0.001 

GM=U2; 

0.00 

0.001 

GM— 313 

0.00 

■i 

0.00 

0.001 

GM=3:14 

0.00 

0.005 

0.00 

0.003 

0.00 

0.002 

0.00 

0.001 

0.00 

0.00 1 

0.00 

0.001 

GM=115 

0.00 

0.005 

0.00 

0.002 

0.00 

0.002 

0.00 

0.001 

0.00 

0.001 

0.00 

0.001 

GM-1 16 

0.00 

0.005 

0.00 

0.002 

0.00 

0.001 

0.00 

0.001 

0.00 

0.001 

0.00 

0.000 

GM=1I7 

0.00 

0.005 

0.00 

0.002 

0.00 

0.001 

0.00 

0.001 

0.00 

0.001 

0.00 

0.000 

GM=118 

0.00 

0.005 

0.00 

0.002 

0.00 

0.001 

0.00 

0.001 

0.00 

0.001 

0.00 

0.000 

GM=119 

0.00 

0.005 

0.00 

0.002 

0.00 

0.001 

0.00 

0.001 

0.00 

0.000 

0.00 

0.000 

GM=120 

i  0.00 

0.005 

0.00 

0.002 

0.00 

0.001 

0.00 

0.001 

0.00 

0.000 

0.00 

0.000 
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Table  4.  Average  (AV G)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  Medium  Difficulty,  Low  Complexity  Condition 


Cut 

Scores 

Alternative  Sample  Sizes 

N  = 

100 

N* 

200 

N- 

400 

N  = 

800 

N  * 

1600 

N* 

3200 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

GM=  80 

0.54 

0.072 

0.55 

0.051 

0.55 

0.037 

0.55 

0.026 

0.55 

0.018 

0,55 

0.013 

GM*  81 

0.54 

0.071 

0.54 

0.051 

0.54 

0.037 

0.54 

0.026 

0,54 

0,018 

0.54 

0,013 

GM-  82 

0.53 

0.070 

0,54 

0.050 

0.54 

0.036 

0.54 

0.025 

0.54 

0.018 

0.54 

0.013 

GM=  83 

0.53 

0.070 

0,53 

0.050 

0,53 

0.036 

0.53 

0.025 

0.53 

0,018 

0.53 

0.013 

GM=  84 

0.52 

0.069 

0.53 

0,049 

0.53 

0.036 

0.53 

0.025 

0.53 

0.017 

0.53 

0.013 

GM-  85 

0,52 

0,068 

0.52 

0.049 

0.52 

0.035 

0.52 

0.024 

0,52 

0.017 

0.52 

0.012 

GM=  86 

0.51 

0,067 

0.52 

0.048 

0.52 

0.035 

0,52 

0,024 

0.52 

0.017 

0.52 

0.012 

GM*  87 

0.51 

0.066 

0.51 

0.047 

0.51 

0.034 

0.51 

0.024 

0.51 

0.017 

0.51 

0.012 

GM*  88 

0.50 

0.065 

0.51 

0.046 

0.51 

0.034 

0.50 

0.023 

0.50 

0.016 

0.50 

0.012 

GM=  89 

0.50 

0,064 

0.50 

0,046 

0,50 

0.033 

0.50 

0.023 

0,50 

0.016 

0.50 

0.012 

GM=  90 

0.49 

0.062 

0,49 

0.045 

0,49 

0.033 

0.49 

0,023 

0.49 

0.016 

0.49 

0.011 

GM=  91 

0.48 

0,061 

0.49 

0.044 

0.49 

0,032 

0.48 

0.022 

0,48 

0.016 

0.48 

0.011 

GM=  92 

0,48 

0.060 

0,48 

0,043 

0,48 

0.031 

0.48 

0,022 

0,48 

0.015 

0.48 

0,011 

GM=  93 

0,47 

0.058 

0.47 

0,042 

0.47 

0.031 

0,47 

0.021 

0,47 

0.015 

0,47 

0,011 

GM*  94 

0,46 

0.057 

0.46 

0.041 

0.46 

0,030 

0,46 

0.021 

0.46 

0.015 

0.46 

0.010 

GM=  95 

0,46 

0.056 

0.46 

0.040 

0.46 

0,029 

0.45 

0.020 

0.46 

0,014 

0.45 

0.010 

GM*  96 

0.45 

0.054 

0.45 

0.039 

0.45 

0.029 

0.45 

0.020 

0.45 

0,014 

0.45 

0.010 

GM=  97 

0,44 

0,053 

0.44 

0,038 

0,44 

0,028 

0,44 

0.019 

0,44 

0.013 

0,44 

0,009 

GM-98 

0.43 

0,052 

0.43 

0.037 

0.43 

0.027 

0.43 

0.018 

0.43 

0.013 

0,43 

0.009 

GM=  99 

0,42 

0.051 

0.42 

0,036 

0.42 

0,027 

0.42 

0.018 

0.42 

0.013 

0.42 

0.009 

GM=100 

0.42 

0,050 

0.41 

0.035 

0.41 

0.026 

0.41 

0.018 

0.41 

0,012 

0.41 

0.009 

GM=101 

0.41 

0.049 

0,41 

0,035 

0,41 

0.025 

0.40 

0.017 

0,40 

0.012 

0.40 

0.008 

GM-102 

0.40 

0.048 

0.40 

0,034 

0.40 

0.025 

0.40 

0,017 

0,40 

0.012 

0.40 

0,008 

GM=103 

0,39 

0.047 

0,39 

0.034 

0.39 

0.025 

0.39 

0,016 

0.39 

0.012 

0.39 

0.008 

GM=104 

0.38 

0.047 

0.38 

0.033 

0,38 

0.024 

0.38 

0.016 

0.38 

0.012 

0.38 

0.008 

GM=i05 

0.37 

0.047 

0.37 

0.033 

0.37 

0.024 

0,37 

0.016 

0,37 

0.012 

0,37 

0.008 

GM*J06 

0,36 

0.047 

0.36 

0,033 

0.36 

0.024 

0.36 

0.016 

0.36 

0.012 

0-36 

0.008 

GM-107 

0.35 

0.047 

0,35 

0,033 

0.35 

0,024 

0.35 

0,016 

0,35 

0.012 

0.35 

0.008 

GM=108 

0.34 

0.047 

0.34 

0.033 

0.34 

0,024 

0,34 

0.016 

0.34 

0.012 

0.34 

0.008 

GM=I 09 

0.33 

0.048 

0.33 

0.034 

0.33 

0.024 

0.33 

0.016 

0,33 

0,012 

0.33 

0.008 

e  4.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  So 
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Table  5.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  Medium  Difficulty,  Medium  Complexity  Condition 


Alternative  Sample  Sizes 


Cut 

Scores 

N- 

100 

N  = 

AVG 

STD 

AVG 

CM-  SO 

0.30 

0,099 

0.31 

CM  81 

0.30 

0.097 

0.30 

GM=  82 

0.29 

0.095 

0.30 

GM=  83 

0.28 

0.093 

0,29 

GM=  84 

0.28 

0.091 

0.28 

GM=  85 

0.27 

0.088 

0,27 

CM-  86 

0.26 

0.085 

0.27 

GM=  87 

0.26 

0.083 

0.26 

GM=  88 

0.25 

0.080 

0.25 

GM=  89 

0.24 

0.077 

0.24 

800 

N  = 

1600 

N  = 

STD 

AVG 

STD 

AVG 

0.038 

0.31 

0,026 

0.31 

0.037 

0.30 

0.026 

0.30 

0.036 

0.29 

0.025 

0,29 

0,035 

0.29 

0.025 

0.29 

0.034 

0.28 

0.024 

0,28 

0.033 

0.27  0.023 

0.27 

0.032 

0.26 

0,023 

0.26 

0.031 

0.26 

— 

0.022 

0.26 

0,030 

0.25 

0.021 

0.25 

0.029 

0,24 

0.020 

0,24 

0,15 

0,010 

0.14 

0.009 

0,13 

CO 

o 

© 

© 

GM=104  !  0.12  0.032  0.12  0.023  0.12  0.016  0.12  0.011]  0.12  0.0081  0.1210.006 

- - - - 1 - - - - - - - 1 - 

GM=105  I  0.12  0.031  0.12 1  0.022  0.12  0.016  0.12 1  0.01 1  I  0.12  0.0 


GM=106  1  0.11  0,030  0.11  0.022  f  0,11  0,015  0.11  0.010 

GM=IQ7  0.10  0.029  0.10  0.021  0.10  0.015  0.10  0.010 

GM=108  0.10  0.029  0.10 1  0.021  0.10  0.014  0.10  0.010 


0,12 

0,006 

0,1 

2 

0,006 

1 

0,005 

0 

0.005 

hu 

Table  5.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  Medium  Difficulty,  Medium  Complexity  Condition 


Alternative  Sample  Sizes 


C»t 

Scores 

N*100: 

N  =  200 

N-t  400 

N  =  800 

N  =  1600 

N  =  3200 

AVG 

STD 

■avg; 

STD 

AVG 

STD 

AVG 

SID 

AVG 

STD 

AVG 

STD 

GM=t()9 

0.09 

0.029 

0.09 

0.020 

0.09 

0.014 

0.09 

0.010 

0.09 

0.007 

0.09 

0.005 

GM=U0 

0.09 

0.028 

0.09 

0.020 

0.09 

0.014 

0.09 

0.010 

0.09 

0.007 

0.09 

0.005 

GM-llI 

0.08 

0.028 

0.08 

0.020 

0.08 

0.014 

0.08 

0.010 

0.08 

0.007 

0.08 

0.005 

GM^lll 

0.08 

0.028 

0.08 

0.020 

0.08 

0.014 

0.08 

0.009 

0.08 

0.007 

0.08 

0.005 

GMN113 

0.07 

0.028 

0.07 

0.020 

0.07 

0.014 

0.07 

0.009 

0.07 

0.007 

0.07 

0.005 

GM=I14 

0.07 

0.028 

0.07 

0.020 

0.07 

0.014 

0.07 

0.009 

— 

0.07 

0.007 

0.07 

0.005 

GM=115 

0.06 

0.028 

0.06 

0.020 

|  0.06 

0.014 

0.06 

0.009 

I  0.06 

0.007 

0.06 

0.005 

GM=116 

|  0.06 

0.029 

0.06 

0.020 

i  0.06 

0.014 

0.06 

0.009 

0.06 

0.007 

0.06 

0.005 

GM=I17 

|  0.06 

;  0.029 

0.06 

0.020 

0.06 

0.014 

0.06 

0.009 

0.06 

!  0.007 

0.06 

0.005 

:gMSll8 

0.05 

0.029 

0.05 

0.019 

0.05 

0.013 

0.05 

0.009 

0.05 

1  0.007 

0.05 

0.005 

GM«119 

0.05 

'  0.029 

0.05 

0.019 

0.05 

0.013 

0.05 

0.009 

0.05 

!  0.007 

0.05 

0.005 

;GM=”.|20 

0.05 

0.029 

0.05 

0.019 

0.05 

0.013 

0.05 

0.009 

0.05 

1 - 

i  0.006 

0.05 

0.005 
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Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD 
and  N  for  Medium  Difficulty,  High  Complexity  Condition 


Alternative  Sample  Sixes 

N  -  100  N  =  200  N  =  400  N  =  800  N.»  1600  | 

AVG  1  STD  AVG  STD  AVG I  STD  AVG [ STD  AVgTsTD  I 


N  -  3200 
AVG  j STD 


0.22 

0.090 

0.21 

0.087 

0.19 

0.083 

0.17  0.075! 


0.16  0.054  0.16  0.038 


0.141  0.035 


IKBBBBI 


0.16  0.027  0.16  0.0191 


0.14 

0.13 

0.12 


0.14  0.018 
0.13  0.016 


0.17  0.015 


0.161  0.014 


.013 
.011 
.12  0.010 


GM-108  0.02  1  0.013 

_ „„ _ _ _ I _ 

GM=109  0.02  0.012 


0.02  0.010  j  0.02  0.007 
0.02  0.009!  0.02  0.006 


0.02  0.0051 
0.02  0.004 


0.02  0.003 


0.02 1  0.003 


0.02  0.002 


0.02  0.002 


Table  6.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  Medium  Difficulty,  High  Complexity  Condition 


Cut  \ 
Scores 

. . . . . . . . . . . . . '  ..  - — T - - - - - — - - - — - - - - - 

,  :  Alternative  Sample  Sizes  ■ .  C 

N-~ 

100 

N  =  200  :l 

N  =  400 

K-800 

N  =  1600 

N  ■=  3200 

AVG 

STD 

AVG 

STI) 

AVG 

STD 

AVG. 

STD 

;avg 

STD 

AVG 

STD 

GM=1I0 

0.02 

— 

0.01 1 

0.02 

0.008 

0.02 

0.006 

0.02 

0.004 

0.02 

0.003 

0.02 

0.002 

GMhJll 

0.01 

0.010 

0.01 

0.007 

0.01 

0.005 

0.01 

0.004 

0.01 

0.003 

0.01 

0.002 

GMM12 

0.01 

0.009 

0.01 

0.007 

0.01 

0.005 

0.01 

0.003 

0.01 

0.002 

0.01 

0.002 

GM=U3 

0.01 

0.008 

0.01 

0.006 

0.01 

0.004 

0.01 

0.003 

0.01 

0.002 

0.01 

0.002 

GMNI14 

0.01 

0.008 

0.01 

0.006 

0.01 

0.004 

0.01 

0.003 

0.01 

0.002 

0.01 

0.001 

GM-llS 

0.01 

i  0.007 

0.01 

0.005 

0.01 

0.004 

0.01 

0.003 

0.01 

0.002 

0.01 

0.001 

GM-116 

0.01 

0.007 

0.01 

0.005 

0.01 

0.003 

0.01 

0.002 

0.01 

0.002 

0.01 

0.001 

GIVI*!!? 

0.01 

0.006 

0.01 

0.004 

0.01 

0.003 

0.01 

0.002 

0.01 

0.001 

0.01 

0.001 

GM=I  18 

0.01 

0.006 

0.01 

0.004 

0.00 

I  0.003 

0.00 

0.002 

0.00 

0.001 

0.00 

0.001 

GM=I19 

0.00 

0.006 

0.00 

0.004 

0.00 

0.002 

0.00 

0.002 

0.00 

0.001 

0.00 

0.001 

GM-120 

0.00 

0.005 

0.00 

0.003 

0.00 

0.002 

0.00 

0.002 

0.00 

|  0.001 

0.00 

0.001 
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Table  7.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  High  Difficulty,  Low  Complexity  Condition 


Cut 

Scores 

GM=  80 
GM=  81 
GM=  82 
GM=  83 
GM=  84 
GM=  85 
GM=  86 
GM=  87 
GM=  88 
GM=  89 

gm=  9o 


N  =  100 
AVG  STD 

0.66  0.058 
0.66  0.057 
0.65  0.057 
0.65  0.057 
0.64  0.056 
0.64  0.056 
0.63  0.055 
0.63  0.055 
0.62  0.0551 
0.62  0.054 
0.61  0.053 
0.61  0.053 


0.60  0.052 


0.59 

0.051 

0.050 

0.58  0.050 


0.049 


0.56  0.04 
0.55  0.04 


0.54 

0.04 

0.52 

0.04 

0.51 

0.04 

0.48  1  0.05 
0.47 1  0.05 


0.46  !  0.036  I  0.46  [  0.02 


Table  7.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  High  Difficulty,  Low  Complexity  Condition 


'Cut  ;  ; 
Scores 

Alternative  Sample  Sixes 

N-  100 

•  .  .  [[L  ^  ••  ••••• 

N-200 

N  -  400 

N-m 

N  ~  1 600 

N-3200 

1 

AVG 

SID 

;avg 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

GM=110 

0.45 

0.053 

0.45 

0.037 

0.45 

0.026 

0.45 

. 

0.018 

0.45 

0.013 

0.45 

0.009 

GM=H1 

0.44 

0.055 

0.44 

0.038 

0.44 

0.027 

0.44 

0.018 

0.44 

0.013 

0.44 

0.009 

GM«I12 

0.43 

0.057 

0.43 

0.039 

0.43 

0.028 

0.43 

0.019 

0.43 

0.013 

0.43 

0.009 

GM=Ti3 

0.42 

0.059 

0.42 

0.040 

0.42 

0.029 

0.42 

0.020 

0.42 

0.014 

0.42 

0.010 

GM--U4 

0.41 

0.061 

0.41 

0.042 

0.41 

0.030 

0.41 

0.020 

0.41 

0.014 

0.41 

0.010 

GM“115 

0.40 

0.063 

0.40 

0.043 

0.40 

0.031 

0.39 

0.021 

0.40 

0.015 

0.39 

0.011 

GM=*116 

1  0.39 

0.066 

0.38 

0.044 

0.38 

0.032 

0.38 

0.022 

0.38 

0.015 

0.38 

0.011 

GM=1 17 

0.38 

0.068 

0.37 

0.046 

0.37 

0.033 

0.37 

0.023 

0.37 

0.016 

0.37 

0.011 

GM*118 

0.37 

0.070 

0.36 

0.048 

0.36 

1  0.034 

0.36 

0.023 

0.36 

0.017 

0.36 

0.012 

GM=119 

0.35 

0.073 

0.35 

0.049 

0.35 

0.035 

0.35 

0.024 

0.35 

0.017 

0.35 

0.012 

GM=120 

0.34 

0.075 

0.34 

0.051 

0.34 

0.036 

0.34 

0.025 

0.34 

0.018 

0.34 

Q.013 
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Table  8.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  High  Difficulty,  Medium  Complexity  Condition 


Cut 

Scores 

1 

j 

Alternative  Sample  Sixes 

N  =  100 

N  =  200- 

N  =  400 

N  -  800 

N  =  1600 

N  —  3200 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

GM=  80 

0.47 

0.078 

0.47 

0,055 

0.48 

0,039 

0,47 

0.027 

0.48 

0,019 

0.48 

0.014 

GM=  81 

0.46 

0.077 

0,47 

0.054 

0.47 

0.039 

0.47 

0.027 

0.47 

0.019 

0.47 

0.014 

GM=  82 

0.46 

0.076 

0,46 

0.054 

0.46 

0,039 

0.46 

0,027 

0.46 

0.019 

0,46 

0,014 

GM=  83 

GM=  84 

0.45 

0.075 

0,46 

0.053 

0.46 

0.038 

0,45 

0.027 

0.46 

0.019 

0.46 

0,013 

0.44 

0.075 

0.45 

0.053 

0,45 

0.038 

0.45 

0,026 

0.45 

0.019 

0,45 

0.013 

GM=  85 

GM=  86 

0.44 

0.074 

0,44 

0,052 

0.44 

0.037 

0.44 

0,026 

0,44 

0.019 

0.44 

0.013 

0.43 

0.072 

0.43 

0,051 

0.43 

0.037 

0.43 

0,026 

0,43 

0,018 

0,43 

0.013 

GM-87 

0.42 

0.071 

0.43 

0.051 

0.43 

0.036 

0.42 

0.025 

0.43 

0.018 

0,43 

0.013 

GM=  88 

0.41 

0.070 

0.42 

0,050 

0.42 

0.036 

0.42 

0.025 

0.42 

0.018 

0.42 

0.013 

GM=  89 

0.41 

0.068 

0.41 

0,049 

0.41 

0,035 

0,41 

0.024 

0.41 

0,017 

0.41 

0.012 

GM=  90 

0.40 

0.067 

0.40 

0,048 

0.40 

0.034 

0,40 

0.024 

0.40 

0,017 

0.40 

0.012 

GM=  91 

0.39 

0.065 

0.39 

0,046 

0.39 

0,033 

0.39 

0.023 

0.39 

0.017 

0,39 

0.012 

GM=  92 

0.38 

0.063 

0,38 

0.045 

0.38 

0.033 

0.38 

0.023 

0.38 

0,016 

0.38 

0.011 

GM=  93 

0.37 

_ 

0,062 

0,37 

0.044 

0.37 

0.032 

0.37 

0,022 

0.37 

0.016 

0.37 

0.011 

GM=  94 

0.36 

0.060 

0,36 

0,043 

0.36 

0.031 

0.36 

0.021 

0,36 

0.015 

0.36 

0.011 

GM=  95 

0.35 

0.058 

0.35 

0,041 

0.35 

0.030 

0.35 

0.020 

0,35 

0.015 

0,35 

0,010 

GM=  96 

0.34 

!  0,056 

0.34 

0.040 

0.34 

0.029 

0.34 

0.020 

0.34 

0.014 

0,34 

0.010 

GM=  97 

0.33 

0.054 

0.33 

0,038 

0.33 

0.028 

0.33 

0,019 

0,33 

0.014 

0.33 

0.010 

GM-98 

0.32 

0,052 

0.32 

0.037 

0.32 

0.027 

0.32 

0.018 

0.32 

0.013 

0,32 

0,009 

GM=  99 

0.31 

0.050 

0.31 

0.035 

0.31 

0.026 

0.31 

0,018 

0,31 

0.012 

0,31 

0,009 

GM=100 

0.30 

0.048 

0.30 

0.034 

0.30 

0.025 

0.30 

0.017 

0.30 

0.012 

0,30 

0.009 

GM=101 

0.29 

0.046 

0.29 

0.033 

0.29 

0,024 

0,28 

0.016 

0.29 

0.011 

0,29 

0.008 

GM=102 

0.27 

0,044 

0.27 

0,032 

0.27 

0.023 

0,27 

0.015 

0.27 

0.011 

0.27 

0,008 

GM=103 

0,26 

0.043 

0.26 

0.031 

0,26 

0,022 

0.26 

0,015 

0.26 

0.011 

0.26 

0.008 

GM=I04 

0.25 

0.042 

0.25 

0.030 

0.25 

0.022 

0.25 

0.014 

0.25 

0.010 

0.25 

0.007 

GM=105 

0.24 

0.041 

0,24 

0.029 

0.24 

0.021 

0.24 

0.014 

0.24 

0.010 

0.24 

0,007 

GM=106 

0,23 

0,040 

0.23 

0,028 

0.23 

0,021 

0,23 

0.014 

0.23 

0.010 

0,23 

0.007 

GM=J07 

0.22 

0,039 

0,22 

0,028 

0.22 

0.020 

0,22 

0.013 

0.22 

0.010 

0.22 

0.007 

GM=108 

0.21 

0,039 

0.21 

0.028 

0.21 

0,020 

0.21 

0.013 

0.21 

0.010 

0,21 

0.007 

GM=109 

0.20 

0.039 

0.20 

0,028 

0,20 

0.020 

0.20 

0,013 

0,20 

0,010 

0,20 

0.007 
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Table  8.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  High  Difficulty,  Medium  Complexity  Condition 


Cut 

Scares 

Alternative  Sample  Sizes 

N« 

too 

N  - 

2&0 

|  "  N  = 

400 

1  :N  = 

800 

N  * 

1600 

N  —  3200 

AVG 

STD 

AVG 

STD 

AVG 

STD 

.AVG: 

STD 

AVG 

STD 

AVG 

STD 

GM=ll0: 

0.19 

0.039 

0.19 

0.028 

0.19 

0.020 

0.19 

0.013 

0.19 

0.010 

0.19 

0.007 

GM-J1I 

0.18 

0.039 

0.18 

0.028 

0.38 

0.020 

0.18 

0.013 

0.18 

0.010 

0.18 

0.007 

G1VI=1 12 

0.17 

0.040 

0.17 

0.028 

0.17 

0.020 

0.17 

0.013 

0.17 

0.010 

0.17 

0.007 

GM-1 1 3 

0.16 

0.040 

0.16 

0.028 

0.16 

0.020 

0.16 

0.013 

0.16 

0.010 

0.16 

0.007 

GM-1 14 

j  0.15 

0.041 

0.15 

0.028 

0.15 

0.020 

0.15 

0.014 

0.15 

0.030 

0.15 

0.007 

GM=U5 

!  0.14 

0.041 

0.14 

0.029 

i  0.14 

0.020 

0.14 

0.014 

0.14 

0.010 

0.14 

0.007 

GM-1 16 

0.14 

:  0.041 

.  0.13 

0.029 

0.13 

0.021 

0.13 

0.014 

0.13 

0.010 

0.13 

0.007 

GM=117 

0.13 

:  0.042 

0.13 

0.029 

0.13 

0.021 

0.13 

0.014 

0.13 

0.010 

0.13 

0.007 

GM=1 18 

0.12 

!  0.042 

0.12 

0.029 

1  0.12 

I  0.021 

0.12 

0.014 

0.12 

0.010 

0.12 

0.007 

GM=1I9 

0.11 

!  0.042 

0.11 

0.029 

0.11 

0.021 

0.11 

0.014 

0.11 

0.010 

0.11 

0.007 

GM=120 

0.11 

0.042 

0.11 

0.029 

0.13 

0.021 

0.11 

0.014 

0.13 

l _ _ 

0.010 

0.10 

0.007 

Table  9.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  High  Difficulty,  High  Complexity  Condition 


Alternative  Sample  Sires 


Cut 

Scores 

N  = 

100 

N  = 

200 

N« 

400 

N- 

800 

N- 

1600 

N  - 

3200 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

AVG 

STD 

CM-  80 

0.42 

0.080 

0.43 

0.055 

0,43 

0.037 

0.43 

0,026 

0.43 

0.019 

0,43 

0,013 

CM-  81 

0.41 

0.080 

0.42 

0.055 

0.42 

0.038 

0,42 

0.026 

0.42 

0,019 

0.42 

0,013 

GM=  82 

0.40 

0.079 

0.41 

0.055 

0,41 

0.038 

0.41 

0,026 

0.41 

0,019 

0.41 

0.013 

CM- 83 

039 

0.079 

0,40 

0,055 

0.40 

0.038 

0.40 

0.027 

0,40 

0.019 

0.40 

0.013 

CM=  84 

038 

0.079 

039 

0,055 

039 

0,038 

039 

0.027 

039 

0.019 

039 

0.013 

CM-  85 

037 

0,078 

038 

0.055 

038 

0,038 

038 

0,027 

038 

0.019 

038 

0.013 

CM-  86 

036 

0.077 

0,37 

0.054 

037 

0,038 

0.37 

0,027 

0.37 

0.019 

037 

0.013 

GM=  87 

0.35 

0.076 

0.36 

0.054 

0.36 

0.038 

0.36 

0.026 

0.36 

0.019 

0.36 

0.013 

CM-  88 

0.34 

0,075 

035 

0.053 

0.35 

0.037 

0.35 

0.026 

0.35 

0.019 

0.35 

0.013 

GM=89 

033 

0.074 

033 

0.053 

033 

0.037 

033 

0.026 

0,33 

0.019 

033 

0,013 

GM=  90 

032 

0.073 

032 

0.052 

0.32 

0.036 

032 

0.026 

0.32 

0.018 

032 

0.013 

GM=  91 

030 

0.071 

031 

0.051 

031 

0.036 

0.31 

0.025 

0.31 

0.018 

031 

0.013 

GM=  92 

0.29 

0.069 

0.29 

0,049 

0.29 

0.035 

0.29 

0.025 

0.29 

0.017 

0.29 

0.012 

CM-  93 

0.27 

0.067 

0,28 

0.048 

0.28 

0,28 

0.017 

0,28 

0.012 

GM=  94 

0.26 

0.064 

0,26 

0.046 

0.26 

BBS 

0.26 

0.016 

0.26 

0.011 

GM=  95 

0.24 

0.061 

0.25 

0,044 

0.25 

0.031 

0.24 

0.022 

0.25 

0.016 

0.25 

0,011 

CM=  96 

0.23 

0.058 

0.23 

0.042 

0.23 

0,030 

0.23 

0,021 

0,23 

0.015 

0.23 

0,010 

CM=  97 

0.21 

0,055 

0.22 

0.040 

0.21 

0,028 

0.21 

0.020 

0.21 

0.014 

0,21 

0.010 

GM=  98 

0.20 

0.052 

0.20 

0.037 

0,20 

0.026 

0.20 

0.019 

0.20 

0.013 

0.20 

0.009 

CM=  99 

0.18 

0.048 

0.18 

0.035 

0.18 

0,024 

0.18 

0,017 

0.18 

0.012 

0.18 

0.009 

GM=100 

0.17 

0,044 

0.17 

0.032 

0.17 

0.023 

0.17 

0.016 

0.17 

0.011 

0.17 

0.008 

GM=I01 

0.15 

0,041 

0,15 

0,030 

0.15 

0.021 

0.15 

0.015 

0,15 

0.010 

0,15 

0,007 

CM- 102 

0.14 

0.037 

0.14 

0,027 

0.14 

0.019 

0.14 

0.014 

0.14 

0.009 

0.14 

0.007 

GM-103 

0.12 

0.034 

0,13 

0.025 

0,12 

0,017 

0,12 

0.013 

0.13 

0.009 

0,13 

0.006 

GM-104 

0.11 

0.032 

0.11 

0.023 

0.11 

0.016 

0.11 

0.012 

0.11 

0.008 

0.11 

0.006 

CM- 105 

0.10 

0.029 

0.10 

0.021 

0.10 

0.015 

0.10 

0.011 

0.10 

0,007 

0.10 

0.005 

CM- 106 

0.09 

0.027 

0,09 

0.019 

0.09 

0.014 

0.09 

0,010 

0.09 

0.007 

0.09 

0.005 

CM- 107 

0.08 

0,025 

0.08 

0.018 

0.08 

0.013 

0.08 

0.009 

0.08 

0.006 

0.08 

0,005 

GM=108 

0.07 

0.024 

0.07 

0.017 

0.07 

0,012 

0.07 

0,009 

0,07 

0.006 

0.07 

0.004 

GM-109 

0.06 

0.022 

0,06 

0,016 

0,06 

0.011 

0.06 

0.008 

0.06 

0.005 

0.06 

0.004 
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Table  9.  Average  (AVG)  Attrition  Rates  and  Standard  Errors  (STD)  by  Cut  Score 
and  N  for  High  Difficulty,  High  Complexity  Condition 


h  Cut  :  \ 
Scores 

Alternative  Sample  Sixes 

N  = 

100 

N  ~ 

200 

N  ;f 

400 

VnA 

800 

N:  = 

1600 

N  “  3200 

AVG 

STD 

AVG 

SID 

AVG 

STD 

AVG: 

STD 

AVG 

STD 

AVG 

STD 

GM-110 

0.05 

0.021 

0.05 

0.015 

0.05 

0.011 

0.05 

0.008 

0.05 

0.005 

0.05 

0.004 

GM-lll 

0.05 

0.020 

0.05 

0.014 

0.05 

0.010 

0.05 

0.007 

0.05 

0.005 

0.05 

0.004 

GM=li2 

0.04 

0.018 

0.04 

0.013 

0.04 

0.009 

0.04 

0.007 

0.04 

0.005 

0.04 

0.003 

•GM«flt3 

0.04 

0.017 

0.04 

0.012 

0.04 

0.009 

0.04 

0.006 

0.04 

0.004 

0.04 

0.003 

GM=114 

0.03 

0.016 

0.03 

0.012 

0.03 

0.008 

0.03 

0.006 

0.03 

0.004 

0.03 

0.003 

GlVi=i!5 

0.03 

0.015 

0.03 

0.011 

0.03 

0.008 

0.03 

0.005 

0.03 

0.004 

0.03 

0.003 

GM=li:6 

0.02 

0.014 

0.02 

0.010 

0.02 

G.0G7 

0.02 

0.005 

0.02 

0.003 

0.02 

0.003 

GM=117 

0.02 

0.013 

0.02 

0.009 

0.02 

0.007 

0.02 

0.005 

0.02 

0.003 

0.02 

0.002 

GM=118 

0.02 

0.013 

0.02 

0.009 

0.02 

0.006 

0.02 

0.004 

0.02 

0.003 

0.02 

0.002 

GM=*J19 

0.02 

0.012 

0.02 

0.008 

0.02 

0.006 

0.02 

0.004 

0.02 

0.003 

0.02 

0.002 

GWMf20 

0.01 

0.011 

0.01 

0.008 

0.01 

0.005 

0.01 

0.004 

0.01 

0.003 

0.01 

0.002 
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APPENDIX  C:  FIGURES 


C-l 


Attrition  Kate  Attrition  Rate 


Average  Attrition  Rates  by  Cut  Score  and  Sample  Size  (N) 


Figure  3.  Results  for  Low  Difficulty,  High  Complexity  Condition 
Attrition  Rates  by  Cut  Score  and  N 
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Figure  4.  Results  for  Medium  Difficulty,  Low  Complexity  Condition 
Attrition  Rates  by  Cut  Score  and  N 
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Figure  7.  Results  for  High  Difficulty,  Low  Complexity  Condition 
Attrition  Rates  by  Cut  Score  and  N 
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Figure  8.  Resuits  for  High  Difficulty,  Medium  Complexity  Condition 
Attrition  Rates  by  Cut  Score  and  N 
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Average  Attrition  Rates  by  Cut  Score  and  Sample  Size  (N) 


Figure  9.  Results  for  High  Difficulty,  High  Complexity  Condition 
Attrition  Rates  by  Cut  Score  and  N 
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Figure  26.  Resuits  for  High  Difficulty,  Medium  Complexity  Condition 
Percent  Reduction  in  Standard  Error  (SE)  by  Cut  Score  and  N 
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Figure  32.  Results  for  Medium  Difficulty,  Medium  Complexity  Condition 
Coefficient  of  Variation  by  Cut  Score  and  N 
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Figure  33.  Results  for  Medium  Difficulty,  High  Complexity  Condition 
Coefficient  of  Variation  by  Cut  Score  and  N 
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Figure  34.  Results  for  High  Difficulty,  Low  Complexity  Condition 
Coefficient  of  Variation  by  Cut  Score  and  N 
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Figure  35.  Results  for  High  Difficulty,  Medium  Complexity  Condition 
Coefficient  of  Variation  by  Cut  Score  and  N 
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Coefficient  of  Variation 


Coefficient  of  Variation  by  Cut  Score  and  Sample  Size  (N) 


Figure  36.  Results  for  High  Difficulty,  High  Complexity  Condition 
Coefficient  of  Variation  by  Cut  Score  and  H 
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Attrition  Rate  Attrition  Rate 


Results  for  Training  Difficulty  (Low,  Medium,  High) 
Average  Attrition  Rates  by  Cut  Score  and  Sample  Size  (N) 

Figure  39.  Results  for  Training  Difficulty  (Low,  Medium,  High) 
Attrition  Rates  by  Cut  Score  for  N=800 
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Figure  40.  Results  for  Training  Difficulty  (Low,  Medium,  High) 
Attrition  Rates  by  Cut  Score  for  N=3200 
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Standard  Error  (SE) 


Results  for  Training  Difficulty  (Low,  Medium,  High) 
Standard  Error  (SE)  by  Sample  Size  (N)  and  Cut  Score 


Figure  41.  Results  for  Training  Difficulty  (Low,  Medium,  High) 
Standard  Errors  (SEs)  by  N  for  GM=85 
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Figure  42.  Results  for  Training  Difficulty  {Low,  Medium,  High) 
Standard  Errors  (SEs)  by  N  for  GM=S5 


Figure  43.  Results  for  Training  Difficulty  {Low,  Medium,  High) 
Standard  Errors  (SEs)  by  N  for  GM=105 
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Results  for  Job  Complexity  (Low,  Medium,  High) 
Average  Attrition  Rates  by  Cut  Score  and  Sample  Size  (N) 

Figure  44.  Results  for  Job  Complexity  (Low,  Medium,  High) 
Attrition  Rates  by  Cut  Score  for  N=100 
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Figure  45.  Results  for  Job  Complexity  (Low,  Medium,  High) 
Attrition  Rates  by  Cut  Score  for  N=400 
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Results  for  Job  Complexity  (Low,  Medium,  High) 
Standard  Error  (SE)  by  Sample  Size  (N)  and  Cut  Score 


Figure  48.  Results  for  Job  Complexity  (Low,  Medium,  High) 
Standard  Errors  (SEs)  by  N  for  GM=85 
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Figure  49.  Results  for  Job  Complexity  (Low,  Medium,  High) 
Standard  Errors  (SEs)  by  N  for  GM=95 
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Figure  50.  Results  for  Job  Complexity  (Low,  Medium,  High) 
Standard  Errors  (SEs)  by  N  for  GM=105 
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Attrition  Rates  with  Standard  Errors  (SE) 
by  Cut  Score  and  N 


Figure  51.  Attrition  Rates  with  Standard  Errors  (SEs) 
by  Cut  Score  for  N=200 
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Figure  52.  Attrition  Rates  with  Standard  Errors  (SEs) 
by  Cut  Score  for  N=800 
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APPENDIX  D:  SAS  PROGRAMS  AND  TECHNICAL  DOCUMENTATION  FOR 

SIMULATION 
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Overview  and  Steps  for  Running  SAS  Programs  for  Simulation 


There  are  five  SAS  programs  associated  with  running  the  simulation  used  in  the  current 
study.  The  names  of  these  SAS  programs,  with  a  brief  description  of  what  each  does,  are  as 
follows  (hard  copies  of  all  programs  appear  on  the  proceeding  pages): 

•  Run_Create_TabIe.sas.  Main  program  for  running  the  simulation.  All  other  programs 
operate  off  of  this  program.  This  is  where  the  configurations  to  run,  location  of  other 
programs  for  running  the  simulation,  plus  location  for  outputting  SAS  datasets  and 
tables,  are  specified. 

•  Create_Table,sas.  Compiles  estimated  regression  parameters  and  activates  the  other 
programs  in  the  simulation  to  generate  the  Word  table  containing  average  attrition 
rates  (and  SEs)  by  sample  size  (N)  and  cut  score. 

•  Simulate_Test_Score_Data_TwoAttempts,sas,  Program  for  generating  synthetic 
training  performance  and  attrition  data.  Sample  size  (. N )  and  number  of  samples  to  be 
replicated  are  based  on  configuration  specifications. 

•  Estimate_Parameter.sas.  Estimates  regression  parameters  to  be  used  as  input  for 
computing  average  attrition  rates  (and  SEs)  at  varying  cut  scores  and  sample  sizes 

(N). 

•  Tabulate_FailRates_FromNormal.sas.  Estimates  average  attrition  rate  (and  SE),  then 
creates  and  outputs  this  information  to  Word  table  for  each  configuration. 

Each  program  contains  documentation  on  the  procedures  and  computations  performed  by  the 
program.  To  run  the  simulation,  follow  these  steps: 

1 .  On  the  computer’s  hard  drive,  create  a  folder  (e.g.,  “SAS  Simulation”)  to  store  the 
programs.  Copy  all  programs  to  this  folder.  There  are  no  constraints  on  the  folder’s 
name,  as  the  programs  are  flexible  enough  to  incorporate  any  name  the  user  specifies, 

2.  Within  the  newly  created  folder  add  a  subfolder  (e.g,,  “Data”).  This  subfolder  will 
contain  the  SAS  datasets  and  tables  outputted  by  the  simulation.  There  are  no 
constraints  on  the  subfolder’s  name,  as  the  programs  are  flexible  enough  to 
incorporate  any  name  the  user  specifies. 

3.  Copy  all  configuration  files  representing  the  different  conditions  into  the  subfolder 
created  in  Step  2.  Do  not  alter  the  names  of  configuration  files,  unless  running  a 
simulation  different  than  the  one  conducted  in  the  current  study. 

4.  Start  SAS.  From  the  “Editor”  window,  go  to  File>Open.  Locate  the  folder 
containing  the  SAS  programs  (created  in  Step  1),  and  select  “Run_Create_Table.sas”. 
When  selected,  hit  “OK”. 
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5.  Follow  the  documentation  contained  in  the  program.  At  a  minimum,  users  will  need 
to  specify  the  location  (folder)  containing  the  SAS  programs  (from  Step  1)  and  the 
folder  to  which  the  SAS  datasets  and  tables  are  to  be  outputted  (from  Step  2). 
Double-check  that  the  location  and  names  of  these  folders  match  those  specified  in 
the  first  two  steps. 

6.  Run  the  program.  To  replicate  the  simulation,  no  other  programs  besides 
“Run_Create_Table.sas”  need  to  be  modified.  All  information  required  by  other 
programs  to  successfully  run  the  simulation  is  contained  in  the 

“Run  Create  Table.sas”  program.  Do  not  make  modifications  to  the  other  programs. 

Please  note  that  replicating  this  simulation  “as  is”,  and  for  all  conditions  (in  a  single  run), 
requires  significant  computer  resources.  Do  not  open  other  programs  when  the  simulation  is 
running.  Users  are  advised  to  plan  on  running  the  simulation  when  computer  resources,  and  the 
time  it  required  to  run  it,  can  be  dedicated  (exclusively)  to  running  the  simulation. 
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Example  of  Configuration  File  (for  Medium  Difficulty, 
Medium  Complexity  Condition) 


[SIMULATION  CONFIGURATION] 

Sample  Size 
5000 

Simulation  Repetitions 
1 

[TEST  CONFIGURATION] 

Number  of  Test 
17 

Minimum  Passing  Score 

80.75  80.75  80.75  80.75  80.75  80.75  80.75  80.75  80.75  80.75  80.75  80.75  80.75  80.75 
80.75  80.75  80.75 
Test  Score  Validities 

.29  .08  .37  .15  .32  .15  .17  .55  .24  .32  .26  .34  .02  .39  .29  .07  .24 
Error  Correlations 
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Number  of  X 
1 

Mean  of  X 
118 

Variance  of  X 
81 

Regression  Weights 

63.56  86.83  62.56  82.93  74.32  80.10  75.46  39.49  68.84  63.04  60.04  68.74  85.61  69.99 
62.45  86.41  79.77 

.24  .06  .24  .10  .18  .11  .11  .43  .17  .23  .25  .18  .02  .22  .24  .03  .13 
Test  Score  Variances 

56.59  46.73  33.03  38.71  24.31  39.04  33.98  49.15  40.34  41.64  74.09  25.77  74.13  25.77 
56.31  21.68  21.65 
Simulation  Random  Number  Seed 
1001 

[REFERENCE  POPULATION] 

Number  of  X 
1 

Mean  of  X 
100 

Variance  of  X 
400 

Alternative  Cut  Score  Range:  LOW,  HIGH,  STEPSI2E 
80  120  1 


Example  of  Configuration  File  (for  Medium  Difficulty, 

Medium  Complexity  Condition) 

FILENAME:  AvgDiff_MedR.Param.txt 

DESCRIPTION:  Test  score  parameters  for  MOS  XX  with  "problem"  data  points  (e.g., 
recruits  below  cut  score,  plus  outlier  on  TestXX)  excluded  from  parameter  estimation. 

NOTE:  Error  structure  does  not  include  negatively  correlated  errors. 

(1)  Training  Difficulty 

*Y-intercepts  are  empirical  estimates  meant  to  reflect  MOS  "average"  in  training 
difficulty  (12-13%  attrit  rate). 

(2)  Test  Validities 

*Test  validities  average  to  .25  ("average")  to  reflect  medium  complexity  MOS. 


Program  A:  Run_Create_Table.sas 

**********************************************************************^^^^^ 
FILENAME :  Run_Create_Table . sas 

***********************************************************************^r***.A.v^.^> 

t 

**********************************************  *  *  ******************************** 
DESCRIPTION;  -  . 

This  program  creates  a  table  with  MEAN  and  STANDARD  ERROR  estimates  of 
expected  attrition/failure  rates  at  alternative  cut  scores  for  specified 
sample  sizes. 

*  ROW  =  Cut  Scores 

*  COLUMN  =  Sample  Size 

*  CELL  =  MEAN  &  STANDARD  ERROR 

NOTE;  ...  . . 

(1)  .  Table,  configuration  is  specified  in  the  input  text  file. 

(2)  Parameter  estimates  are  stored  separately  by  sample  size  and  replication. 

****★***************************************************************************. 

t 

******************************************************************************** 
STEP  1;  SET  GENERAL  PARAMETERS 

*  mDataDir  specifies  location  of  directory  containing  parameter  configuration 
files  and  to  which  SAS  dataset s,  output  &  log  files,  and  tables  are  outputted. 

*  SET  mDropTest=YES  to  drop  Test  Score  and  only  keep  ATTRIT .  Else,  set  to  NO. 

********************************************************************************. 

t 

%let  mDataDir=D: \SAS  5imulation\Data; 

%let  mDropTest^YES; 


******************************************************************************** 
STEP  2:  SET  PROGRAMS  DIRECTORY  LOCATION 

*  mProgDir  specifies  location  of  directory  containing  SAS  programs. 

*  ■fr*****************************************************************************.*^ 

/ 

%let  mProgDir=D; \SAS  Simulation; 

************************  *  *  ****************************************************** 

STEP  3:  SET  TITLE  OF  OUTPUT  TABLE  (OPTIONAL) . 

******  +  **********************************  +  ***  +  ******.********.********.*.****.******.*, 

%let  titlelinel  "This  is  title  line  #1”; 

%let  titleline2  "This  is  title  line  #2"; 

%let  titleline3  "This  is  title  line  #3”; 

******************************************************************************** 
STEP  4:  RUN  PROGRAM 

*  mDataFile  specif ies ' name  of  all  configuration  files,  SAS  datasets,  and  RTF 

files  read  by  and  outputted  by  the  simulation.  NOTE:  This  name  reflects  key 

configuration  parameters  (i.e.,  level  of  difficulty  and  complexity). 

***************************************  *'*  **************************************** 

/ 

options  formchar=f | “++++++*  nodate  nonumber  mprint; 

/**Medium  Difficulty,  Medium  Complexity  Condition**/ 

%let  mDataFile^AvgDif fJMedR; 

% include  " &mProgDir \Create_Table .sas" ; 

/**Medium  Difficulty,  High  Complexity  Condition**/ 

%let  mDataFile==AvgD±ff_HighR; 

%include  "&mProgDir\Create_Table . sas" ; 
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Program  A:  Run_Create_Table.sas 


/ * *Medium  Difficulty,  ..Low  Complexity  Condition**/ 
%let  rriDataFile=AvgDif f_LowR; 

% include  " SmProgDir \Cr eat e_T able . sas " ; 

/**Low  Difficulty,  Medium  Complexity  Condition**/ 
%let  mDataFile=LowDif f_MedR; 

%include  "&mProgDir\Create_Table . sas" ; 

/**Low  Difficulty,  High  Complexity  Condition**/ 
%let  mDataFile=LowDif f_HighR; 

%include  "&mProgDir\Create_Table . sas " ; 

/**Low  Difficulty,  Low  Complexity  Condition**/ 

%let  mDataFile==LowDif  f_LowR; 

%include  "&mProgDir\Create_Table . sas " ; 

/**High  Difficulty,  Medium  Complexity  Condition**/ 
%let  mDataFile=HighDif f_MedR; 

%include  "&mProgDir\Create_Table . sas” ; 

/**High  Difficulty,  High  Complexity  Condition**/ 
%let  mDataFile=HighDif f_HighR; 

%include  "&mProgDir\Create_Table .  sas"; 

/**High  Difficulty,  Low  Complexity  Condition**/ 
%let  mDataFile=HighDif f_LowR; 

%include  "&mProgDir\Create_Table . sas"/' 
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FILENAME :  Create_Table . sas 

************************  *****************************i'*it  +  +  ir±±±±*if  +  ±  +  +  +  ir±  +  i'i'  +  *  +  j'9 

$ 


*****************  **  ***************************************^^.^^.^r^r^^^if^.^^^.^^^,^^ril.^^e 
DESCRIPTION:  Compiles  estimated  regression  parameters  and  activates  other 
programs  (e.g.,  Simulate_Test_Score_Data_TwoAttempts . sas,  Estimate_Parameter , sas) 
for  generating.  Word  table  reporting,  the  estimated  average  attrition  rate  (and 
standard  error)  by  cut  score  and  sample  size. 

NOTE:  Run  this  program  through  RUN_CREATE_TABLE . SAS .  Do  NOT  modify  this  program. 

**************************************************ieif*^^r^r^^!r^e.^^r^^,^t^r^^r^^^^^ir^^^;.^^c^^^f9 

/ 

******************************************************1f±  +  *i(±±ii:±*i'iir  +  ±*i'*  +  +  +  ic**±j'jf* 

*  mColFile  specifies  name  of  SAS  dataset  containing  logistic  regression 
parameters  by  sample  size  across  replications;  NOTE:  Name  of  file  reflects 
mDataFile  specification  from  Run__Create_Table. sas. 

*■.  mOutFile  specifies  name  of  SAS  dataset  "containing  attrition  rates  by  cut  scores 
across  replications.  NOTE:  This  file  is  written  in  mDataDir  directory.  Name  of 
file  reflects  mDataFile  specification  from  Run_Create_Table. sas. 

*  mRtf File  specifies  name  of  the  RTF  file  version  of  the  OUTPUT  TABLE.  This  file 
can  be  imported  directly  to  MS  Word.  NOTE:  This  file  is  written  in  mDataDir 
directory.  Name  of  file  reflects  mDataFile  specification  from 
Run_Create_Table . sas , 

****** **********************************************************************.****. 
%let  mColFile=&mDataFile .__LogistParam; 

%let  mOutFile=&mDataFile._Table; 

%let  mRtf File=&mDataFile. .rtf ; 

proc  printto  print="toDataDir\&mDataFile ._Output . txt" 
log=”&mDataDir\&mDataFile . . log"; run; 

*********************************************************  +  +  +  +  + 

(1)  Reads  in  parameters  from  configuration  file. 

*********************************************************±**  +  *  +  ±ieiemfir*±+4r  +  ±±  +  je*.*' 

data  _null_; 
file'  print; 
length  strTmp  $  100; 

infile  "SmDataDir . \&mDataFile . . Param. txt "  lrecl-100  TRUNCOVER; 
do  until  (trim (strTmp) =” [SIMULATION  CONFIGURATION]"); 
input  strTmp  $100.; 
put  strTmp; 

end; 


nNumSize  =0; 
input  strTmp  $100.; 
do  i-1  to  10  until (nSize=. ) ; 
input  nSize  @; 
put  "NSIZE  =  "  nSize; 

nNumSize  =  nNumSize  +  1; 
if  (nSizeA=.)  then 

call  symput  (compress  ("mSize"|  I  put  (i,  2.).) ,  left  (put  (nSize,  5. )))  ; 

end; 

nNumSize  —  nNumSize  -  1; 

put  "Number  of  Sample  Sizes  =  ”  nNumSize; 
call  symput { "mNumSize”, put (nNumSize,  2. ) ) ; 


Input  /  StrTmp  $100 . ; 
input  nReps; 
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put  "NREPS  =  "  nReps ; 

call  .symput  ( "mNumRep", put  (nReps,  5.  )  )  ; 

stop; 
run  ; 

%macro  MacroCreateTable; 

*******************************  *******************************.^llr.*..A,<Ar.}t.Ar.)t,A..jt.£r.Ar,£..jt.A.  +  <Ar,Ar 

(2)  Creates  the  simulated  test  score  data. 
**********************************************^***^^  +  ^^^^^  +  +  ^  +  iJ|fS(r^^^  +  ^  +  +  ^^^^^^  +  ^^^ 

%do  iSize  =  1  %to  &mNumSize; 

%let  mNumSSN  =  &&mSize&iSize; 

%  include  "  &mProgDir \Simulate__Test_Score_Data__TwoAttempts  .  sas  "  ; 

%end; 

★  ************************************************************^.A.^  +  ^<A>^  +  +  ^  +  +  ^^ 

(3)  Computes  the  parameter  estimates  across  replications. 
***************************************************************.^  +  +  it  +  +  +  ^^^^^ 

%include  " &mProgDir\Est imate__Parameter .  sas " ; 

***************************************************************.A,  +  +  +  vk.  +  ^  +  ^  +  ^  +  +  +  +  + 

(4)  Generates  the  attrition  table. 

************************  ********************************************************. 
%include  f,&mProgDir\Tabulate_FailRates_FromNormal .  sas" ; 

%mend; 

IMacroCreateTable ; 

**************************************************************++^+^++^*+^++++^+ 

* 

END  OF  PROGRAM. 

**************** ****************************************************************, 

/ 
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Program  C:  Simulate_Test_Score_Data_TwoAttempts.sas 


********************************************************  + 

FILENAME:  SimulateJTest_ScoreJ)atajrwoAttempts . sas 
***************************************  *****  +  *******^  +  +  ^  +  ^^^^  +  it^  +  ^  +  ^  +  +  it^  ^ 


DESCRIPTION:  Program  for  generating  synthetic  training  performance  and  attrition 
data , 

NOTE:  Run  this  program  through  RUN_CREATE_TABLE . SAS .  Do  NOT  modify  this  program. 

****************je*********jtjt*****  +  *ie4r*j(jir*jcjeit*4f^it*ie*itjtjr*1c1cic****i'’k'k'k'k'k-k'k*'k-k-k-k-jc~kiejc-fr 

t 


libname  DataDir  "&mDataDir"; 

filename  ParFile  "SmDataDirX&mDataFile . .Param.Txt"; 


proc  iml; 


**************************  ****************************************************** 
READING  IN  TEST  CONFIGURATION  INFORMATION  FROM  INPUT  PARAMETER  FILE. 

*  macro  variable  =  SmParFile 


(1)  Test  score  data  dimension. 

*  nNumTest  -  scalar  number  of  tests 

*  nNumSSN  -  scalar  number  of  SSNs  (Sample  Size) 

*  nNumRep  -  scalar  number  of  simulation  repetitions 

(2)  Minimum  passing  scores  for  school  tests,  which  can  vary  across  tests. 

*  fMinPassScore  -  vector  of  nNumberTest  elements 

(3)  "Restricted"  test  validities. 

*  fRxy  -  vector  of  nNumberTest  validities.  Equals  corr (X, Y[l:nNumberTest] ) 

(4)  Intra-person  test  score  "error"  correlations.  Allow  unequal  correlations. 

*  fRho  —  symmetric  matrix,  nNumberTest*nNumberTest  intra— person  error 
correlations. 

(5)  Mean  and  variance  of  predictor  matrix  X. 

*  nNumX  -  scalar  number  of  predictors,  including  constant 

*  fMuX  -  test  score  means 

*  fVarX  -  test  score  variances 

(5)  Regression  constants  and  coefficients  for  test  score  means  conditional  on  X. 

*  fBeta  -  (p+1)  by  (nNumberTest)  matrix.  Columns  represent  tests.  Note  that 

E{ Y[ j ] IXvec}  =  t (Xvec) *fBeta [, j] 

(6)  Variance  of  tests  (restricted  to  MOS  school  sample) 

*  fVarY  -  1  X  nNumTest  vector  of  test  variances 

********************************************** it********************************, 

infile  ParFile; 


nNumSSN  =  SmNumSSN; 

print  nNumSSN [label="Sample  Size"]; 

nNumRep  =  SmNumRep;  . 

print  nNumRep [label="Number  of  Repetitions"]; 

do  until  (trim(strLabel)=" [TEST  CONFIGURATION]"); 

input  strLabel  $RECORD.; 
end;  -  . 


input  strLabel  $ RECORD.  /  nNumTest; 
print  nNumTest [label=strLabel] ; 

input  strLabel  $  RECORD .  ; . - . . 

fMinPassScore  =  repeat (0, 1, nNumTest) ; 
do  j=l  to  nNumTest; 
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input  tmpVal  @; 

^  -fMinPassScore [ j ]  =  tmpVal; 
end; 

print  fMinPassScore [label=strLabel] ; 
input; 

input  strLabel  $ RECORD . ; 
f Rxy  =  repeat (0,1, nNumTest ) ; 
do  j=l  to  nNumTest; 
input  tmpVal  @; 
fRxy[j]  =  tmpVal; 

end; 

print  fRxy [label=strLabel] ; 
input; 

•input  strLabel  $ RECORD . ; 

“fRho  =  repeat (0, nNumTest, nNumTest ) ; 
do  i=l  to  nNumTest; 
do  j=l  to  nNumTest; 
input  tmpVal  @; 

fRho[i,j]  =  tmpVal; 

end; 

input; 

end; 

print  fRho [ label=strLabel] ; 

input  strLabel  $ RECORD .  /  nNumX; 
print  nNumX [label=strLabel] ; 

input  strLabel  $ RECORD . ; 
fMuX  =  repeat (0, 1, nNumX) ; 
do  j=l  to  nNumX; 
input  tmpVal  @; 
fMuX[j]  =  tmpVal; 

end; 

print  fMuX [label=strLabel] ; 
input; 

input  strLabel  $ RECORD. ; 
fVarX  =  repeat (0, nNumX, nNumX) ; 
do  i=l  to  nNumX; 
do  j=l  to  nNumX; 
input  tmpVal  @; 

fVarX [ i , j ]  =  tmpVal; 

’  end; 

input; 

end; 

print  fVarX [label=strLabel] ; 

•input  strLabel  $ RECORD . ; 
fBeta  =  repeat (0, nNumX+1, nNumTest) ; 
do  i=l  to  nNumX+1; 
do  j=l  to  nNumTest; 
input  tmpVal  @; 

- fBata  [i,  j  =  tmpVal; 

end; 

input; 
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end; 

print  fBeta [label=strLabel] ; 

input  strLabel  $RECORD. ; 
fVarY  =  repeat (0, 1, nNumTest) ; 
do  j=l  to  nNumTest; 
input  tmpVal  @; 

f Va  rY [ j ]  =  tmpVal ; 

end; 

print  fVarY [label=strLabel] ; 

input  /  strLabel  $RECORD.  /  nRandSeed; 
print  nRandSeed [label=strLabel]  ; 


*  w  T  1 


CREATING  OUTPUT  FILE  FOR  SYNTHETIC  TRAINING  PERFORMANCE  AND  ATTRITION  DATA. 

*  FILEREF  =  DataDir . DataFile 

*HiH*ttHHHi*H****iHHtH*HHtHHt***H**HHt*H***4******HHH***HHt. 

strColNames  =  compress (concat ("X",  char (0 :nNumX) )) ; 
if  ( "&mDropTest,,A="YES")  then  do; 

strColNames  =  strColNames 1  I  compress (concat ( "SC0RE1_",  char (1 : nNumTest ))) ; 
strColNames  =  strColNames 1 | compress (concat ( "SCORE2_",  char (1 : nNumTest ))) ; 
end; 

strColNames  =  strColNames | | "ATTRIT"; 

create  DataDir. &mDataFile._N&mNumSSN  var  ( "REP" | | strColNames ) ; 


GENERATING  SYNTHETIC  DATA  THAT  REFLECTS  CONFIGURATION  PARAMETERS  (e.g.,  MEAN, 
VARIANCE,  etc.) 

****************************** **************************************************. 

/ 


/*  Cholesky  of  variance  of  X  */ 
rootVarX  =  root(fVarX); 

/*  "n  XI"  unit  vector  */ 
oneXVec  =  repeat (l,nNumSSN, 1) ; 


/*  Root  of  error  variances  */ 
mseScore  =  ( (l-fRxy##2) IfVarY) ## . 5; 

/*  Error  covariance  */ 

errorVar  =  diag (mseScore) *fRho*diag (mseScore) ; 
/*  Cholesky  of  error  covariance  */ 
rootVarError  =  root (errorVar); 


do  iRep=l  to  nNumRep; 

V*  Simulating  training  performance  over  two  attempts  */ 

/*  Simulating  samples  of  AA  composite  scores  of  N  size  (i.e.,  nNumSSN)  with 
.  mean  and  variance  approximately  equal  to  mean  and  variance  of  real-world  data. 
*/ 

fXmat  =  oneXVec  ||  oneXVec*fMuX  +  rannor (repeat (1, nNumSSN, nNumX) ) *rootVarx; 

/*  Testl  and  Test2  are  samples  of  simulated  test  scores  with  mean,  variance, 
validities,  etc.,  approximately  equal  to  real-world  estimates.  */ 
fTestl  =  fXmat*fBeta  + 

rannor (repeat (nRandSeed, nNumSSN,  nNumTest) KrootVarError; 
fTest2  =  fXmat*fBeta  + 

rannor (repeat (nRandSeed,  nNumSSN,  nNumTest) ) *rootVarError; 
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Identifying  max  test  score  across  two  attempts  to  compute  attrition  */ 
f TestMax  =  fTestl<>fTest2; 

/*  If  mDropTest  set  to  "Yes",  dropping  test  score  data,  but  keeping  attrition 
data  to  minimize  file  size.  Otherwise,  all  data  (test  scores,  attrition) 
outputted  to  SAS  dataset.  */ 

■if  ( " &mDropTest " A=" YES " )  then  do;  . -  -  ..  . 

OUTMAT  =  repeat (iRep, nNumSSN, 1) | | 
fXmat  |  | 

fTestl  |  | 

“  ‘  f Test2  |  | 

/*  Computing  attrition  (1="ATTRIT",  0="NO  ATTRIT" )  based  on  max  test 
score  and  minimum  passing  scores  * / 

(fTestMax  <  oneXVec*fMinPassScore) [ , <>] ; 

end; 

else  do; 

OUTMAT  =  repeat (iRep, nNumSSN, 1) | | 
fXmat  | | 

/*  Computing  attrition  (1="ATTRIT",  0="NO  ATTRIT")  based  on  max  test 
score  and  minimum  passing  scores  */ 

(fTestMax  <  oneXVec*fMinPassScore)  ( ,  <>] ; 

end; 

append  from  OUTMAT; 
end; 

close  DataDir. &mDataFile ._N&mNumSSN; 

quit; 

run; 


END  OF  PROGRAM.' - -  ' '  ' 


Program  D;  Estimate_Parameter.sas 


★  *********■&************  •k0’k  ******************************************* fc************ 

FILENAME:  Estimate_Parameter . sas 

***************** r***************************************************************B 

/ 


****************************************************£******************  +  **^  +  ^^ 
DESCRIPTION:  Program  for  estimating  regression  parameters ,  which  become  input 
for  estimating  average  attrition  rates  and  standard  errors  by  sample  size  and 
cut  score. 

NOTE:  Run  this  program  through  RUN_CREATE_TABLE . SAS .  Do  NOT  modify  this  program. 

************************************ *************************************+******# 

t 


/*  Macro  for  compiling  individual  SAS  datasets  containing  synthetic  data  for  a 
given  sample  size  {across  X  replications} .  */ 

%macro  Set Files;  ... 

%do  iSize  =  1  %to  SmNumSize; 

DataDir . kmDataFile ,_N&&mSize&iSize  (in=In&&mSize&iSize) 

%end; 

%mend; 

/*  Macro  for  computing  sample  sizes  to  be  included  in  TestScoreData.  */ 

Imacro  NSizes; 

In&mSizel*&mSizel 

%do  iSize  —  2  %to  &mNumSize; 

+  In&&mSize&i3ize*&&m3ize&iSize 
%end; 

%mend; 

/*  Merging  individual  SAS  datasets  containing  synthetic  data  for  a  given  sample 
size  (across  X  replications)  into  single*  omnibus  dataset  with  variable  NSIZE.  */ 
data  TestScoreData  / view=Test SeoreDat a ; 
length  NSIZE  4; 
set  %SetFiles; 

NSIZE  =  INSizes; 
run; 

/*  Estimating  regression  parameters  for  each  replication  (REP)  by  sample  size 
(NSIZE).  Outputting  parameters  to  specified  SAS  dataset.  */ 
proc  logistic  data=TestScoreData 

outest=datadir . SmColFile  (keep=NSIZE  REP  Intercept  XI) 
descending  noprint; 
model  attrit  =  xl; 
by  NSIZE  rep; 
quit; 
run; 

******************************************************************************** 
END  OF  PROGRAM. 

*******************************************************  *************************  . 

* 
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************  *************************************** 
■  FILENAME:  -  Tabulate_FailRate.s_FromNormal.sas 
+***************************************+**^**^^+^ 


DESCRIPTION:  Estimates  average  attrition  rate  (and 
and  cut  score,  then  creates  table  and  outputs  this 
•NOTE:  Run -this-  program  through  RUN_CREATE_TABLE .  SAS 
+*+*+******************************************^^+ 


standard  error)  by  sample  size 
information  to  RTF  file. 

.  Do  NOT  modify  this  program. 

***************************** , 

9 


libname  DataDir  "&mDataDir"; 

filename  'RowFile  M&mDataDir\&mDataFile  .  . Param. Txt " ; 


-proc  iml; 


*  * 


READING  IN  POPULATION  PARAMETERS  AND  CUT  SCORE  RANGE  FOR  ESTIMATING  ATTRITION 
RATES. 

*  macro  variable  =  &mRowFile 


(1)  Mean  and  variance  of  predictor  matrix  X  in  REFERENCE  POPULATION. 

*  nNumX  -  scalar  number  of  predictors,  including  constant. 

*  fMuX  -  reference  population  mean  for  AA  composite. 

*  fVarX  -  reference  population  variance  for  AA  composite. 

(2)  Alternative  Cut  Scores  along  the  row  dimension. 

*  nNumCut  -  total  number  of  cut  scores. 

*  fXCut  -  list  of  cut  scores  for  which  attrition  rates  (and  SEs)  will  be 
estimated. 


* . 
t 


infile  RowFile; 


do  until  (trim (strLabel) =" [REFERENCE  POPULATION]"); 

input  strLabel '$ RECORD. ; 
end; 

input  strLabel  $ RECORD .  /  nNumX; 
print  nNumX [label=strLabel] ; 

/*  Reading  in  reference  population  mean.  */ 
input  strLabel  $ RECORD. ; 
fMuX  =  repeat (0, 1, nNumX) ; 
do  j=l  to  nNumX; 
input T tmpVal  @; 
fMuX [ j ]  =  tmpVal; 

..  end; 

print  fMuX [label=strLabel] ; 

'/■*•  Reading  in  reference  population  variance.*/ 
input ; 

input  strLabel  $ RECORD . ; 
fVarX  =  repeat ( 0, nNumX, nNumX) ; 
do  i=l  to  nNumX ; 

-■*  do  -  j^-1  ..to  .  nNumX; 

input  tmpVal  @; 

fVarX [i,j]  ~  tmpVal; 
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end; 

. input; 

end; 

print  fVarX [label=strLabel] ; 

/*  Reading  in  specified  cut  score  range.  */ 
input  strLabel  $ RECORD. ; 
input  fCutLO  fCutHI  fCutSTEP; 
print  fCutLO  fCutHI  fCutSTEP; 

nNumCut  *  floor  { (fCutHI-fCutLO)  /fCutSTEP)  +  Im¬ 
print  nNumCut  [label=="Number  of  Cut  Scores"]; 
fXCut  =  repeat (0, 1,  nNumCut ) ; 
do  j=l  to  nNumCut; 

fXCut [ j ]  -  fCutLO  +  ( j  —1 ) *fCutST£P; 

end; 

print  fXCut [label=strLabel] ; 

/*  Copy  LO,  HI,  STEP  to  macro  variables  for  later  processing.  */ 
call  symput ("mCutLO", compress (char (fCutLO) ) ) ; 
call  symput ("mCutHI",  compress  (char (fCutHI ) ) ) ; 
call  symput ("mCutSTEP", compress (char (fCutSTEP) } ) ; 

******************************************************************************** 
CREATING  OUTPUT  DATASET  CONTAINING  ESTIMATED  ATTRITION  RATES  BY  SAMPLE  SIZE  AND 
CUT  SCORE. 

*  FILEREF  =  DataDir . &mOut File; 

********** *********************************************************************** 

/ 


strColNames  =  compress (concat ( "FATTRIT", char (fXCut) ) ) ; 
create  DataDir . &mOutFile  var  ({"NSIZE"  "REP"}  M  strColNames); 

******************************************************************************** 
ESTIMATING  ATTRITION  RATES  BY  SAMPLE  SIZE  AND  CUT  SCORE. 

*  FILEREF  =  DataDir . SmColFile; 

*************************************  *  * *  * *  *  ******  *  ***  *  *  * *  *  *  *  *  * * *  *****  ************ 

t 


nNumQuant  =  5000; 

/*  Generating  quantiles  based  on  reference  population  mean  and  variance  */ 
quantX  =  probit ( (1 : (nNumQuant-1) ) /nNumQuant ) *sqrt (fVarX [ 1, 1] )  +  fMuX; 

/*  Creating  nNumQuant  BY  nNumCut  matrix  of  0  and  1  to  denote  pass  cut  score  */ 
aboveCut  =  repeat  (t  (quantX)  ,  1,  ncol  (fXcut )  )  >  repeat  (fXcut,  ncol  (quantX)  ,  1)  ; 

strBetaNames  =  "Intercept"  ||  compress (concat ( "X", char  ( 1  :nNumX)  ))  ; 

u'se  DataDir .  &mCol File; 

/*  Looping  through  sample  size  and  replications  */ 
do  data; 

read  next  var  {"NSIZE"  "REP"); 

— read  current  var  (strBetaNames)  into  fBeta; 

/*  Estimating  probability  that  recruits  within  a  quantile  would  not 
complete  training  */ 

quantProbAttrit  =  1/(1  +  exp  (-fBeta  [1]  -  quantX*  fBeta.[  2]  }  )  ; 

'/*  Calculating  attrition  rate  for  each  alternative  cut  score  in  specified 
range.  For  given  alternative  cut  score,  the  attrition  rate  is  the  simple 
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arithmetic  average  of  probabilities  of  not  completing  (i.e., 
quantProbAttrit)  associated  with  AA  quantiles  above  the  cut  score.  This  is 
equivalent  to  average  of  probability  of  not  completing  weighted  by  density 
of  AA.  */ 

cutProbAttrit  =  NSIZE  ||  REP  ||  (quantProbAttrit*aboveCut) /aboveCut [+,] ; 
append  from  cutProbAttrit; 

end; 

close  DataDir . SmColFile; 

quit; 
run ; 

*********  *********************************************************************** 
CREATING  AND  OUTPUTTING  RTF  FILE  CONTAINING  TABLE  WITH  ESTIMATED  ATTRITION 
RATES  BY  SAMPLE  SIZE  AND  CUT  SCORE. 

*  FILEREF  =  DataDir . SmOutFile  AND  &mDataDir\&mRtf File; 
******************************************************************************** 

/*  Macro  for  creating  sample  size  labels  for  table.  */ 

Imacro  SizeLabel; 

%do  iSize=l  %to  &mNumSize; 

&&mSize&iSize="N  =  &&mSize&iSize" 
lend; 

%mend; 

/*  Macro  for  creating  cut  score  variables  needed  to  build  table.  */ 

%macro  CutScoreVars ; 

%do  i=&mCutLO  %to  &mCutHI  %by  &mCutSTEP; 

PATTRIT&i 

%end; 

%mend; 

/*  Macro  for  creating  cut  score  labels  for  table.  */ 

%macro  CutScoreLabel; 

%do  i=&mCutLO  %to  100  %by  SmCutSTEP; 

PATTRIT&i ;=  MGM=  &i” 

%end; 

Ido  i=100  % t o  &mCutHI  %by  SmCutSTEP; 

PATTRIT&i  =  "GM=&i" 
lend; 

Imend; 

proc  formate- 

value  nsizefmt  % SizeLabel ; 
run  ; 

/*  Creating  and  outputting  table  containing  average  (AVG)  attrition  rates  and 
standard  errors  (STD)  by  sample  size  and  cut  score.  */ 
ods  rtf  f ile-M&mDataDir\&mRtfFileM ; 
proc  tabulate  data=DataDir . &mOutFile; 
titlel  &titlelinel; 
title2  &titleline2; 
title 3  &titleline3; 
class  NSIZE; 
format  NSIZE  nsizefmt.; 
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Program  E;  Tabulate_FailRates_FromNormal.sas 


var  %Cut ScoreVars  ; 

table  %CutScoreVars/,  .  . 

NSIZE* (M£AN*F=6.2  STD*F=6.3) 
/box="Cut  Scores"  rts=20; 
keylabel  mean="AVG" 
std  =”5TD”; 

label  NSI2E="Alternative  Sample  Sizes" 
ICutScoreLabel ; 
run;  ; 

ods  rtf  close; 


END  OF  PROGRAM. 
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APPENDIX  E:  SAS  PROGRAMS  AND  TECHNICAL  DOCUMENTATION  FOR 
PERFORMING  ATTRITION  RATES  ANALYSIS 
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Overviewand  Steps  for  Running  SAS  Programs  for  Attrition  Rates  Analysis 


Only  a  single  SAS  program  is  needed  to  perform  an  attrition  rates  analysis, 
Tabulate_Attrit  Rates_ForSchool.sas  (a  hard  copy  of  this  program  appears  on  the  proceeding 
pages).  The  program  contains  documentation  on  the  procedures  and  computations  performed  by 
the  program,  plus  the  specifications  users  need  to  make  in  order  to  run  it.  The  program  was 
designed  to  minimize  the  number  of  specifications  users  must  make  in  order  to  run  the  program. 
Users  can  easily  customize  these  specifications  to  fit  their  analysis  needs.  In  general,  to  run  an 
attrition  rates  analysis,  follow  these  steps: 

1.  On  the  computer’s  hard  drive,  create  a  folder  (e.g.,  “SAS  Simulation”).  This  folder 
will  contain  the  SAS  program,  and  all  datasets  and  tables  outputted  by  the  program. 
There  are  no  constraints  oft  the  folder’s  name,  as  the  program  is  flexible  enough  to 
incorporate  any  name  the  user  specifies. 

2.  Copy  the  program  (Tabulate_Attrit  Rates_ForSchool,sas)  and  the  SAS  dataset 
containing  the  MOS  school  data  to  be  analyzed  to  this  folder.  Prior  to  running  the 
analyses,  be  sure  that  the  integrity  of  the  MOS  school  data  has  been  verified; 
suggestions  for  doing  so  were  made  in  the  “Discussion”  section  of  this  report.  At  a 
minimum  check  the  integrity  of  AA  composite  scores  and  attrition  data,  as  the 
attrition  rates  analysis  primarily  utilizes  these  data.  Note,  that  the  variable  containing 
attrition  data  should  be  a  dichotomous,  numeric  variable  (e.g.,  l=”attrit”,  0=”no 
attrit”),  with  non-academic  attritions  classified  as  “missing.”  It  is  important  that  all 
non-academic  attritions  are  excluded  from  the  analysis. 

3.  Start  SAS.  From  the  “Editor”  window,  go  to  File>Open.  Locate  the  folder  created  in 
Step  1,  and  select  “Tabulate_Attrit  Rates_ForSchool.sas.”  When  file  has  been 
selected,  click  “OK”. 

4.  Follow  the  documentation  contained  in  the  program.  At  a  minimum,  users  will  need 
to  specify: 

•  The  location  (folder)  containing  the  SAS  program  and  dataset  with  MOS 
school  data  (created  in  Step  1); 

=■  ,  i.Ihe  name  of  the  SAS- dataset  containing  the  MOS  school  data  to  be 

analyzed; 

•  The  variables  names  (in  the  SAS  dataset)  for  the  AA  composite  and 
attrition  data; 

•  The  mean  and  variance  of  the  AA  composite  (applicable  to  the  MOS  to  be 
analyzed)  for  the  appropriate  reference  population;  and 

-  •  ■  The  range  of  cut  scores-  for  which  attrition  rates  should  be  estimated. 
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5.  After  the  above  have  been  specified  (and  any  additional  options,  such  as  titles  for  the 
table),  run  the  program.  All  information  required  to  successfully  perform  the  attrition 
rates  analysis  is  contained  at  the  top  of  the  “Tabulate_Attrit  Rates_ForSchool.sas” 
program.  Users  do  not  need  to  make  modifications  to  the  rest  of  the  program. 

The  table  outputted  by  the  program  is  comparable  to  the  table  produced  by  the  simulation,  except 
the  table  is  composed  of  only  two  columns.  The  first  column  (“Cut  Scores”)  lists  the  range  of 
alternative  cut  scores,  as  specified  by  the  user.  The  second  column  (“Attrit”)  reports  the 
corresponding  estimated  attrition  rates.  An  example  table  appears  after  the  copy  of  the  program. 
The  program  outputs  the  table  to  an  .RTF  file  that  can  be  easily  opened  in  MS  Word. 


Tabulate  AttritRates  ForSchool.sas 


*************************  *  ******  *  *  *  *  *  *  *  *  *  *  *  *  ************  *  *****  *  *  *  *  ************  * 
FILENAME ;  Tabulate_AttritRates__ForSchool .  sas 

************************************************************,£***^***^*^,;t,t.A.^,£,A.,jr,ifi 

/ 

******************************************************************************* 

DESCRIPTION:  Program  for  estimating  attrition  rates  by  cut  score  using  available 
MOS  school  data. 

***********************************  *  *  ******************************************. 

f 

*****  **************  ************************************************************* 
STEP  1A:  SET  FILE  LOCATIONS. 

The  following  parameters  are  required.  They  must  be  modified  in  order  to  run  the 
program  successfully. 

*  mDataDir  specifies  location  of  directory  containing  SAS  datasets, 
configuration  files,  etc.,  used  by  program,  and  to  which  all  SAS  datasets  and 
tables  generated  by  program  are  outputted. 

*  mDataFile  specifies  name  of  SAS  dataset  containing  MOS  school  data.  NOTE:  By 

default,  name  of  dataset  becomes  first  part  of  filename  for  all  files 

generated  by  the  program. ■ SEE  mColFile,  mOutFile,  and  mRtfFile. 

****************  *  *  *  *  *  *  *'*'**  *  *  *  *  *  *  *  *  *-*  *  *  *  ***************************************** 

f 

%let  mDataDir=D: \SA3  Simulation\Data; 

%let  mDataFile=MOS55d; 

******************************************************************************.** 
STEP  IB:  SET  FILE  LOCATIONS  (con,) 

NOTE:  The  following  parameters  are  optional.  They  do  not  have  to  be  modified 
unless  desired  by  the  user. 

*  mColFile  specifies  name  of  SAS  dataset  containing  logistic  regression 
parameters.  NOTE:  The  default  filename  is  mDataFile  +  " ._LogistParam" . 

*  mOutFile  specifies  name  of  SAS  dataset  containing  attrition  rates  by  cut 
scores.  The  file  becomes  input  to  the  Word  table  produced  by  this  program. 

NOTE:  The  default  filename  is  mDataFile  +  "._Table" . 

*  mRtfFile  specifies  name  of  the  RTF  file  version  of  the  table  containing 

estimated  attrition  rates  by  cut  score.  This  file  can  be  imported  opened  in 

Word.  NOTE:  The  default  filename  is  mDataFile  +  M.rtf". 

************************* ******************************************************* 

f 

%let  mColFIle=&mDataFile  *_LogistParam; 

%let  mOutFile^&mDataFile  ._Table; 

%let  mRtfFile^&mDataFlle. . rtf ; 

************ ******************************************************************** 
STEP  2:  SPECIFY  VARIABLE  NAMES  IN  SCHOOL  DATASET. 

*  mAAComp  specifies  the  variable  name  of  the  AA  composite  (e.g.,  AR,  GM,  MK,  AS, 
etc.)  used  to  determine  enlistment  eligibility.  NOTE:  This  information  will 
also  be  used  to  label  cdt  adores  in  the "outputted  table  (e.g.,  GM=80r  GM=82r 
G"M=84 , "  etc. }  . 

*  mAttrit  specifies  the  variable  name  containing  academic  attrition  information. 

******************************************************************************** 

t 

%let  mAAComp=GM; 

%let  mAttrit=attrit ; 

******************************************************************************** 
STEP  3:  SET  REFERENCE  POPULATION  PARAMETERS  AND  CUT  SCORE  RANGE. 

*  fMuX  specifies  the  reference  population  mean  for  the  AA  composite. 

*  fVarX  specifies  the  reference  population  variance  for  the  AA  composite. 

*  fCutLO  defines  the  lower  bound  of  the  range  of  cut  scores  for  which  attrition 
rates  will  be  estimated. 


E-4 
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*  fCutHI  defines  the  upper  bound  of  the  range  of  cut  scores  for  which  attrition 
rates  will  be  estimated. 

*  fCutSTEP  specifies  which  cut  scores  within  the  defined  range  of  cut  scores 
(see  fCutLO  and  fCutHI)  attrition  rates  will  be  estimated  for  (and  will  appear 
in  the  outputted  table) .  The  value  specified  here  defines  the  increment  in  the 
cut  score  range. 


■***★•***★* 


■%let  fMuX=100; 
%let  fVarX=400; 
%let  fCutLO=80; 
%let  fCutHI=120; 
%let  fCutSTEP=l ; 


**  *  *  * 


STEP  4:  SET  TITLE  OF  OUTPUT  TABLE  (OPTIONAL). 

%let.  titlelinel  "This  is  title  line  #1"; 

%let  titleline2  "This  is  title  line  #2"; 

%let  titleline3  "This  is  title  line  #3"; 


*****  +  *****  +  +  *  +  *  +  *  +  ***  +  **it**  +  *  +  ****±ie*****  +  ie**  +  +i'*  +  ic-k  +  -ir'k‘k*'k'k-k*  +  'ie'k  +  'k**'k*'k*-k-k*1'-)c-)' 

START  OF  PROGRAM.  DO  NOT  MODIFY  SAS  CODE  BEYOND  THIS  POINT. 

************************************************************************.A.  +  *^  +  +> 

r 

libname  DataDir  "&mDataDir"; 


************* ******************************************************************* 
COMPUTING  REGRESSION  PARAMETERS  TO  ESTIMATE  ATTRITION  RATES  AT  DIFFERENT  AA 
COMPOSITE  CUT  SCORES. 

*  FILEREF  =  DataDir . SmDataFile  AND  Datadir . SmColFile; 

**************************************************:(r^rilf  +  +  +  +  ifc  +  +  llf  +  ^.  +  +  +  A  +  sk>it^>  +  +  +  ^  +  it  +  ^^ 

r 

proc- logistic  data=DataDir . &mDataFile 

outest=Datadir . &mColFile  (keep=  Intercept  &mAAComp) 
descending  noprint; 
model  &mAttrit  =  SmAAComp; 
quit; 
run; 

***********  ***************************************************************_A.**  +  ^ 

READING  IN  POPULATION  PARAMETERS  AND  CUT  SCORE  RANGE  FOR  ESTIMATING  ATTRITION 
RATES. 

"  ^  ^  ^  ^  A"  ^  "jc  ^  ^  ^  ^  'A  ^  St"  ^  ^  ^  ^  ^  ^  ^  ^  ^  ^  ^  ^  ^  ^  ^  ^  ^  ^  ^ 


proc  iml; 

/*  Reading  in  reference  population  parameters  and  cut  score  range  from  macro 
above.  */ 
fMuX=&fMuX; 
fVarX=&fVarX; 
fCutLO=& fCutLO; 
fCutHI=&fCutHI; 

•f CutSTEP=& fCutSTEP ; 

print  fMuX [label="Population  Mean  of  AA  Composite"]; 
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T  abulate_AttritRates_F  orSchool .  sas 

print  fVarX[label="Population  Variance  pi  AA  Composite"]; 
print  fCutLO  fCutHI  fCutSTEP; 

nNumCut  =  floor { (fCutHI-fCutLO) /fCutSTEP)  +  1; 
print  nNumCut  [label="Number  o'f  Cut  Scores"]; 
fXCut  =  repeat (0,1, nNumCut ) ; 
do  j=l  to  nNumCut; 

fXCut [j]  =  fCutLO  +  Xj-1) * fCutSTEP; 

end; 

print  fXCut [label="Alternative  Cut  Scores"]; 

/*  Copy  LO,  HI,  STEP  to  macro  variables  for  later  processing.  */ 
call  symput ("mCutLO",  compress (char (fCutLO) )) ; 
call  symput  ( "mCutHI",  compre'ss  (char  (fCutHI ) )  ) ; 
call  symput ("mCut STEP” , compress (char (fCutSTEP) ) ) ; 

i***************************************************************^**.^*,^.^^.^ 

CREATING  OUTPUT  DATASET  CONTAINING  ESTIMATED  ATTRITION  RATES  BY  CUT  SCORE. 

*  FILEREF  =  DataDir, SmOutFile; 

************************************_*******************************************. 

0 

strColNames  =  compress (concat ("PATTRIT", char (fXCut) )) ; 
create  DataDir. SmOutFile  var  (strColNames); 

**************************************  ****************  ************************* 
ESTIMATING  ATTRITION  RATES  BY  CUT  SCORE 

*  FILEREF  =  DataDir* SmColFile; 

*******************************************************************************. 
nNumQuant  =  5000; 

/*  Generating  quantiles  based  on  reference  population  mean  and  variance  */ 
quantX  =  probit ( (1: (nNumQuant-1 ) ) /nNumQuant) *sqrt (fVarX[l, 1] )  +  fMuX; 

/*  Creating  nNumQuant  BY  nNumCut  matrix  of  0  and  1  to  denote  pass  cut  score  */ 
aboveCut  =  repeat (t (quantX) , l,ncol (fXcut) )  >  repeat (fXcut, ncol  (quantX) , 1) ; 

strBetaNames  =  "Intercept"  | |  "SmAAComp"; 

use  DataDir. SmCol File; 

read  var  (strBetaNames)  into  fBeta; 

/*  Estimating  probability  that  recruits  within  a  quantile  would  not 
complete  training  */ 

quantProbAttrit  =  1/(1  +  exp (-fBeta [1]  -  quantX*fBeta [2] ) )  ; 

/*  Calculating  attrition  rate  for  each  alternative  cut  score  in  specified 
range.  For  given  alternative  cut  score,  the  attrition  rate  is  the  simple 
’  arithmetic- average  of  probabilities  of  not- -completing  (i.e., 

■  •  quantProbAttrit)  associated  with  AA  quantiles  above  the  cut- score.  This  is 
equivalent  to  average  of  probability  of  not  completing  weighted  by  density 
of  AA.*/ 

cutProbAttrit  =  (quantProbAttrit* aboveCut) /aboveCut [  +  , ]  ; 

append  from  cutProbAttrit; 

close  DataDir . SmColFile; 

quit; 

run; 
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CREATING  AND  OUTPUTTING  RTF  FILE  CONTAINING  TABLE  WITH  ESTIMATED  ATTRITION  RATES 
BY  CUT  SCORE. 


*  FILEREF  =  DataDir . SmOutFile  AND  &mDataDir\&mRtf File; 


/*  Macro  for  creating  cut  score  variables  needed  to  build  table.  */ 
%macro  CutScoreVars ; 

%do  i=&mCutLO  %to  SmCutHI  %by  SmCutSTEP; 

PATTRIT&i 

%end; 

%inend; 


/*  Macro  for  creating  cut  score  labels  for  table.  */ 
%macro  CutScoreLabel; 

%do  i=&mCutLO  %to  100  %by  &mCutSTEP; 

PATTRIT&i  =  "  &mAAComp=  &i" 

%end; 

%do  i=100  % to  SmCutHI  %by  SmCutSTEP; 

PATTRIT&i  =  "&mAAComp=&i,, 

%end; 

%mend; 


/*  Creating  and  outputting  table  containing  estimated  attrition  rates  by  cut 
score.  */ 

ods  rtf  f ile=M &mDataDir\&mRt f File" ; 
proc  tabulate  data=DataDir . &mOutFile; 
titlel  &titlelinel; 
title2  &titleline2; 
title3  &titleline3; 
var  %CutScoreVars; 
table  %CutScoreVars , 

MEAN*F=6 . 2 

/box="Cut  Scores”  rts=20; 
keylabel  mean=”Attrit ” ; 
label  %CutScoreLabel ; 
run  ; 

ods  rtf  close; 


END  OF  PROGRAM. 


Example  of  Table  Produced  by  Attrition  Rates  Analysis 

Table  1.  Estimated  Attrition  Rates  by  Cut  Score  for  MOS  XX  (N=XXX) 


igqMp 
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Attrit 

GM=  85 

0,27 

0.26 

GM=  87 

0.25 

GM=  88 

0.25 

GM=  89 

0.24 

■ :  '  •  " 

0.23 
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0.22 

0.21 
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0.21 
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0.20 
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0.19 
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GM=  97 

0.18 

GM=  98 

0.17 

GM=  99 
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0.16 
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0.15 

GM=101 

0.15 

■ 

0.14 

GM=103 

0.13 

GM=104 

0.13 
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0.12 
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